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Preface 


The  Operations  Analysis  Program  of  the  Office  of  Naval  Research  (ONR)  sponsored 
this  work  to  apply  a  research  and  development  (R&D)  portfolio  management  decision 
framework  recently  developed  by  the  RAND  Corporation  to  the  evaluation  of  ONR 
applied  research  projects.  This  framework  computes  the  Expected  Value  of  an  R&D 
project  as  the  product  of  three  factors:  value  to  the  military  of  the  capability  sought 
through  R&D,  the  extent  to  which  the  performance  potential  matches  the  level  required 
to  achieve  the  capability,  and  the  projects  transition  probability.  The  objective  of  the 
framework  is  to  maximize  the  benefit  of  R&D  spending  in  the  presence  of  uncertainty 
inherent  to  R&D.  The  objective  of  this  project  was  to  demonstrate  the  framework 
through  a  case  study  evaluation  of  sample  ONR  Code  3 1  (Information,  Electronics, 
and  Surveillance  Department)  projects  in  command,  control,  and  communications. 

This  report  should  be  of  interest  to  all  stakeholders  in  the  Navy  R&D  process,  as 
well  as  R&D  sponsors  and  managers  throughout  the  military  and  civilian  sectors. 

This  research  was  conducted  for  ONR  within  the  Acquisition  and  Technology 
Policy  Center  of  the  RAND  National  Defense  Research  Institute,  a  federally-funded 
research  and  development  center  sponsored  by  the  Office  of  the  Secretary  of  Defense, 
the  Joint  Staff,  the  unified  COMMANDS,  and  the  defense  agencies. 

For  more  information  on  RAND  s  Acquisition  and  Technology  Policy  Center,  con¬ 
tact  the  Director,  Philip  Anton.  He  can  be  reached  by  email  at  AT_PC-Director@rand. 
org;  by  phone  at  (310)  393-0411,  extension  7798;  or  by  mail  at  RAND,  1700  Main 
Street,  Santa  Monica,  California  90407-2138.  More  information  about  RAND  is 
available  at  www.rand.org. 


Contents 


Preface . iii 

Figures . *x 

Tables . xi 

Summary . xlil 

Acknowledgments . xlx 

Abbreviations . ™ 

CHAPTER  ONE 

Introduction . 1 

ONR  Responsibility  for  Three  Different  Navies . 1 

The  Challenge  of  Use-Inspired  Basic  Research . 2 

The  Challenge  of  Facilitating  Transition . 2 

Closing  the  Basic  Research-Technology  Development  Gap  with  6.2  Research . 2 

The  RAND  PortMan  R&D  Portfolio  Management  Decision  Framework . 5 

Unique  Aspects  of  the  RAND  Decision  Framework . 6 

Using  Expert  Judgment  to  Estimate  the  Expected  Value  Components . 7 

Role  of  USNR  on  the  Project  Team . 7 

Outline  of  the  Report . 8 

CHAPTER  TWO 

Background . 9 

Department  of  the  Navy  6.3  Review . 9 

The  ONR  Code  31  Program  Evaluation  Process . 10 

Related  R&D  Evaluation  Approaches . 12 

CHAPTER  THREE 

Methods . 14 

Overview  of  the  RAND  PortMan  Portfolio  Management  Decision  Framework . 14 

Capability . lb 


vi  Contents 


Performance  Potential . 17 

Transition  Probability . 17 

Value  Descriptions  and  Scales . 17 

Data  Requirements . 70 

Overview  of  the  RAND  E-DEL+I  Technique  and  Process . 22 

Comparison  of  E-DEL+I  and  Traditional  Consensus  Building  Methods . 23 

The  E-DEL+I  Process . 73 

Previous  RAND  Experience  Using  the  E-DEL+I  Method . 25 

CHAPTER  FOUR 

Case  Study  Description . 77 

Composition  of  the  Expert  Panel . 77 

The  Sample  ONR  Code  31  6.2  Research  Projects . 27 

The  Assessment  Dimensions . 78 

The  Assessment  Questionnaire . 79 

General  Info . 79 

Relevance . 79 

Applicability . 30 

Achievement . 30 

Conditions . 30 

Status . 30 

Implementability . * . 30 

Summary . 30 

Reference . 30 

Feedback . 30 

RdXText . 31 

Dynamics  of  the  Evaluation . 31 

Expert  Panel  Concerns  and  Final  Assessments . 32 

Summary  of  Consensus  Positions . 33 

CHAPTER  FIVE 

Analysis  of  the  Case  Study  Results . 36 

Expected  Value  Ranges  of  the  Projects . 36 

The  Components  of  Expected  Value . 38 

Capability  versus  Performance  Potential . 38 

Capability  versus  Transition  Probability . 41 

Performance  Potential  versus  Transition  Probability . 42 

Demonstration  of  a  Portfolio  Analysis . 44 

CHAPTER  SIX 

Conclusions  and  Recommendations . 51 


Contents  vii 


Appendix  A.  DoD  Definitions  of  R&D  Categories . 55 

Appendix  B.  The  Island  Defense  Scenario . 57 

Appendix  C.  Value  Descriptions  and  Scales . 62 

Appendix  D.  Project  Summary  Format . 65 

Appendix  E.  Project  Overviews . 66 

Bibliography . 73 


Figures 


S.  1  Portfolio  View  of  Case  Study  Evaluation  Results . xv 

1.1  Schematic  View  of  the  Transition  from  Basic  Research  Investment  to  Meeting  Navy 

and  Marine  Corps  Requirements . 3 

1 .2  Schematic  View  of  the  Navy  R&D  Process . 4 

3. 1  Portfolio  View  of  the  RAND  PortMan  Decision  Framework . 1 5 

3.2  E-DEL+I  Consensus  Formation  Process . 24 

4. 1  Screenshot  from  E-DEL+I  Excel  Workbook . 31 

4.2  Modes  and  Consensus  Positions  for  Capability/Relevance . 34 

5. 1  Expected  Value  Ranges  of  the  Case  Study  Projects . 37 

5.2  Capability  versus  Performance  Potential . 39 

5.3  Capability  versus  Performance  Potential  with  Uncertainty . 40 

5.4  Capability  versus  Transition  Probability . 42 

5.5  Performance  versus  Transition  Probability  (versus  Fig  5.4) . ...43 

5.6  Expected  Value  Portfolio  Analysis  by  Quadrant . 44 

5.7  Expected  Value  Portfolio  Analysis  by  Grouping . 46 

5.8  Range  of  Uncertainty  for  Group  1 . 47 

5.9  Range  of  Uncertainty  for  Group  2 . 47 

5.10  Range  of  Uncertainty  for  Group  3 . 48 

5.11  Range  of  Uncertainty  for  Group  4 . 48 

5.12  Range  of  Uncertainty  for  Group  5 . 49 


Tables 


3.1  Capability . 18 

3.2  Performance  Potential . 19 

3.3  Transition  Probability . 19 

3.4  Capability  Scale . 20 

3.5  Performance  Potential  Scale . 20 

3.6  Transition  Probability  Scale . 20 

3.7  Project  Summary  Format . 21 

4. 1  Representation  of  Navy  E-DEL+I  Panel  of  Experts . 28 

4.2  Case  Study  Project  Tides  and  Abbreviations . 29 

4.3  Summary  of  Consensus  Positions  of  the  Expert  Panelists . 35 


XI 


Summary 


The  Office  of  Naval  Research  (ONR)  has  the  responsibility  for  defining  and  sponsor¬ 
ing  research  and  development  (R&D)  necessary  to  support  both  the  current  and  future 
requirements  of  the  Navy  and  Marine  Corps.  To  accomplish  this  mission  ONR  must 
fund  a  broad  spectrum  of  research,  ranging  from  basic  research  needed  to  open  up  new 
options  for  the  long-term,  to  very  near-term  advanced  technology  development  to  sup¬ 
port  the  current  fleet.  Moreover,  ONR  must  make  its  R&D  funding  decisions  in  the 
presence  of  uncertainty:  uncertainty  in  required  capabilities,  uncertainty  in  performance 
requirements,  and  uncertainty  in  the  feasibility  of  a  technology  or  R&D  approach. 

This  report  describes  the  adaptation  of  an  R&D  portfolio  management  decision 
framework  recently  developed  by  RAND  (Silberglitt  and  Sherry,  2002),  PortMan,  to 
support  ONR’s  R&D  decision-making,  and  the  demonstration  of  its  use  via  a  case 
study  evaluation  of  20  sample  ONR  applied  research  projects. 


RAND's  PortMan  R&D  Portfolio  Management  Decision  Framework 

RANDs  PortMan  R&D  decision  framework  computes  the  ExpectedValuex  of  an  R&D 
project  as  the  product  of  three  factors:  value  to  the  military  of  the  capability  sought 
through  R&D,  the  extent  to  which  the  performance  potential  matches  the  level  required 
to  achieve  the  capability,  and  the  projects  transition  probability.  A  useful  interpreta¬ 
tion  of  this  equation  is  that  the  performance  potential  (i.e.,  the  performance  level  that 
would  be  achieved  if  the  project  were  successful),  scales  the  capability  value  that  is 
based  on  achieving  a  required  level  of  performance.  That  scaled  capability  is  then  fur¬ 
ther  reduced  by  the  transition  probability  in  order  to  obtain  the  expected  value  of  the 
research,  including  its  subsequent  R&D  stages  and  the  fielding  of  the  resulting  process, 
component,  or  system. 


1  The  equation  for  Expected  Value  used  in  PortMan  is  based  on  an  adaptation  of  Decision  Theory  (see,  e.g.,  RaifFa, 
1997)  as  described  in  detail  in  Silberglitt  and  Sherry  (2002). 
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PortMan  does  not  rely  on  the  expected  value  as  a  point  solution.  Rather,  it  includes 
an  estimate  of  uncertainty,  and  evaluates  and  compares  R&D  projects  based  upon  all 
three  components  of  the  expected  value,  including  uncertainty,  plus  their  estimated 
direction  of  change  over  time.  This  requires  evaluation  based  on  best  current  informa¬ 
tion,  and  tracking  over  time  as  conditions  change  and  the  technology  matures. 

We  note  that  the  purpose  of  PortMan  is  to  evaluate  a  defined  group  of  actual 
or  proposed  projects  and  to  provide  a  means  for  creating  a  portfolio  from  them  that 
maximizes  the  value  of  R&D  investments.  It  does  not  generate  an  absolute  score  for 
the  total  portfolio  that  could  be  used  to  compare  to  portfolios  of  other  projects  or  to 
proportionally  allocate  funds  between  portfolios  of  different  projects. 

An  important  feature  of  this  approach  is  that  it  allows  identification  of  those 
R&D  projects  seeking  high-value  capabilities,  but  with  formidable  technical  or  field¬ 
ing  problems  remaining  to  be  solved,  and  for  which  management  attention  may  have 
the  greatest  leverage. 

The  case  study  application  of  PortMan  used  RANDs  E-DEL+I™  collaborative 
consensus  building  method  (Wong,  2003)  with  an  expert  panel  of  Naval  officers  and 
senior  civilian  executives.  This  method  retains  the  independent  assessment  features  of 
the  traditional  Delphi  method  (Dalkey,  1969),  and  adds  a  teleconference  between  the 
participants  to  provide  verbal  feedback,  in  addition  to  the  traditional  written  feedback. 
This  case  study  evaluation  was  conducted  via  email  and  teleconference  between  partici¬ 
pants  in  California,  Rhode  Island,  Virginia,  and  Washington,  D.C. 

It  is  important  to  keep  in  mind  that  the  evaluation  described  in  this  report  is  only 
a  case  study  application  of  the  RAND  PortMan  R&D  portfolio  management  decision 
framework.  As  such,  it  was  performed  within  the  time  and  budget  constraints  of  the 
project  and  should  not  be  regarded  as  a  complete  or  accurate  evaluation  of  the  20  sam¬ 
ple  R&D  projects  evaluated.  It  is,  however,  valid  as  a  case  study  demonstration  of  the 
use  of  the  framework,  because  all  of  the  evaluators  had  exactly  the  same  data  on  which 
to  base  their  evaluations,  and  the  RAND  E-DEL+I  consensus  generation  method  was 
consistently  applied  using  the  decision  frameworks  anchored  scales,  together  with  a 
representative  and  well-qualified  expert  panel. 


Portfolio  Analysis  and  Investment  Strategy 

Figure  S.l  summarizes  the  results  of  the  case  study  evaluation  of  20  sample  ONR 
applied  research  projects.  It  plots  the  product  of  the  mean  values,  as  determined  by 
the  expert  panel,  of  capability  and  performance  potential  versus  the  mean  values  of 
transition  probability.  The  expected  value  for  each  project  is  shown  in  parentheses  next 
to  its  data  point.  Contours  of  equal  expected  value  are  shown  as  a  percentage  of  the 
maximum  expected  value. 

This  method  of  combining  the  components  of  expected  value  provides  a  conve¬ 
nient  way  to  identify  R&D  projects  with  similar  characteristics  from  an  investment 
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Figure  S.1 

Portfolio  View  of  Case  Study  Evaluation  Results 


Transition  probability 

RAND  MG271-S.1 


strategy  perspective.  For  example,  consider  the  groupings  shown  in  Figure  S.l.  Group 
1,  the  highest  on  both  axes,  are  the  projects  that  are  most  likely  to  provide  high-value 
capabilities  that  can  be  transitioned  to  use  by  the  Navy.  Group  2  are  projects  that  are 
likely  to  be  transitioned,  but,  because  of  their  relatively  low  values  on  the  y-axis,  are 
also  likely  to  provide  only  incremental  improvements  in  current  capabilities.  Group 
3  are  projects  with  the  potential  to  provide  high-value  capabilities,  but  which  are  not 
likely  to  transition  without  some  changes  to  current  approaches  and  plans.  Group  4 
are  projects  that  appear  to  be  aimed  at  incremental  capabilities  and  do  not  have  strong 
transition  plans.  Group  5  are  projects  that  have  not  yet  formed  their  transition  strate¬ 
gies;  one  of  these  is  aimed  at  a  highly-valued  capability,  while  the  other  is  not. 

As  shown  in  Chapter  5  (see  Figures  5.8  through  5.12),  these  project  groupings  are 
consistent  with  the  uncertainty  indicated  by  the  spread  of  judgments  obtained  from 
the  expert  panel,  in  that  the  mean  values  for  each  group,  plus  the  range  of  uncertainty, 
occupies  a  separate  area  of  the  plot. 

The  project  groupings  in  Figure  S.l  identify  projects  for  which  the  R&D  manager 
may  have  the  greatest  leverage.  Consider,  for  example,  the  projects  in  Group  3,  which 
are  currently  ranked  relatively  high  in  value  and  in  the  intermediate-  to  low-range  in 
transition  probability.  In  order  to  be  successful,  these  projects  will  have  to  develop  and 
implement  a  transition  plan.  We  posit  that,  absent  attention  by  the  Program  Manager, 
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the  Principal  Investigators  may  fail  to  do  this  within  the  time  needed  for  a  successful  tran¬ 
sition.  Or,  if  they  are  not  constrained  sufficiendy  by  floors  on  performance  objectives, 
their  tendency  may  be  to  seek  out  and  pursue  the  easiest  possible  transition  opportuni¬ 
ties.  This  will  increase  their  transition  probability,  but  will  likely  happen  at  the  expense  of 
lowered  performance  objectives,  which  will  move  them  toward  (the  incremental)  Group 
2  Managements  job  in  this  case  is  to  provide  the  resources  (e.g.,  staff,  funds,  facilities, 
information,  coordination,  planning)  to  overcome  either  of  these  tendencies  and  push 
the  projects  toward  Group  1,  as  opposed  to  Group  2,  maintaining  or  even  improving 
the  capability  and  performance  potential  objectives,  while  improving  the  transition  pros¬ 
pects.  Chapter  5  suggests  the  following  set  of  steps  to  develop  an  investment  strategy  that 
can  accomplish  this  task. 

1.  Review  the  data,  consider  the  reasons  for  large  uncertainties  and  outliers,  and 
note  any  effects  on  project  positions; 

2.  Give  a  high  priority  to  providing  sufficient  resources  for  each  Group  1  project  to 
achieve  its  objectives; 

3.  Review  the  transition  plans  of  the  projects  in  Group  2  and  allocate  funds  based 
on  the  relative  importance  of  their  capabilities,  based  on  user  inputs; 

4.  Review  the  technical  approach  and  objectives  of  the  projects  in  Group  3,  then 
provide  the  resources  to  develop  and  implement  transition  plans  that  will  allow 
the  achievement  and  fielding  of  their  identified  capabilities; 

5.  Look  for  additional  opportunities  for  projects  in  Groups  4  and  5  that  have  highly 
valued  capabilities  and  for  which  the  same  approach  as  above  in  4  can  be  pur¬ 
sued. 

6.  Fund  remaining  Group  4  and  Group  5  projects  based  on  the  value  of  their  capa¬ 
bilities,  if  and  only  if  a  sound  case  can  be  made  for  their  moving  into  Group  2  on 
a  specific  time  frame  consistent  with  program  objectives. 

Since  R&D  decisions  are  made  under  uncertainty,  it  is  critical  to  periodically  recon¬ 
sider  the  analysis  and  decisions  based  on  the  latest  and  best  updated  available  data. 
In  this  sense,  the  S&T  metric  of  RANDs  PortMan  decision  framework  is  the  posi¬ 
tion  of  each  project  on  the  chart  of  Figure  S.l,  together  with  its  uncertainty,  moni¬ 
tored  over  time.  As  the  framework  is  repeatedly  used,  and  the  steps  above  repeatedly 
performed,  this  metric,  and  estimates  of  its  direction  of  motion,  will  constitute  a 
dynamic  means  for  maximizing  the  leverage  of  program  funds  to  develop  and  transi¬ 
tion  highly  valued  capabilities.  We  note  that  when  using  this  metric,  it  is  crucial  to 
either  continue  with  substantially  the  same  expert  panel,  or  to  use  a  control  group 
or  perform  a  detailed  analysis  of  variation  to  ensure  that  changes  in  the  evaluations 
result  principally  from  changes  in  the  projects  or  the  requirements  and  not  from 
changes  in  the  panel. 
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Conclusions  and  Recommendations 

As  described  in  detail  in  Chapters  1-5,  RAND  successfully  adapted  PortMan  from 
civilian  to  military  use.  The  case  study  evaluation  demonstrated  the  use  of  the  frame¬ 
work  to  develop  and  analyze  the  data  needed  to  perform  an  R&D  portfolio  analysis 
and  develop  an  R&D  investment  strategy,  including  uncertainty. 

This  framework  can  be  used  with  an  expert  group  determining  the  values  of  capability, 
performance  potential,  and  transition  probability,  including  uncertainty.  In  the  case  study 
evaluation,  this  was  accomplished  with  RAND  s  E-DEL+I  consensus  building  method. 

The  framework  is  capable  of  comparing  and  contrasting  individual  and  groups 
of  research  projects  as  a  function  of  key  management  variables  such  as  their  potential 
value  and  status  and  quality  of  transition  planning.  It  also  provides  a  straightforward 
and  logical  set  of  repeatable  steps,  (as  described  in  Chapter  5)  based  on  auditable 
data  (e.g.,  developed  for  the  case  study  as  described  in  Chapter  4),  to  determine  the 
expected  value  of  research  projects,  together  with  a  measure  of  uncertainty. 

The  framework  provides  a  set  of  parameters  for  each  research  project  (capability, 
performance  potential,  transition  probability)  that,  together  with  measures  of  their 
uncertainties,  form  a  metric  that  can  be  monitored  over  time  as  the  framework  is 
applied  iteratively. 

Based  on  the  success  of  the  case  study  evaluation,  and  the  illustrations  of  its  use  for 
portfolio  analysis  and  investment  strategy  development  presented  in  Chapter  5,  we  recom¬ 
mend  that  ONR  take  the  next  step  of  demonstrating  the  framework  on  a  specific  project 
portfolio. 

One  possibility  might  be  to  incorporate  the  framework  into  an  existing  project 
review  process,  such  as  that  currendy  used  by  ONR  Code  31,  as  described  in  Chapter 
2.  In  this  case,  capability  value  scales  would  be  based  on  a  group  of  scenarios  or  mission 
capability  packages  identified  in  collaboration  with  ONR  Code  31,  the  value  deter¬ 
minations  would  be  made  by  the  existing  Code  31  expert  panel,  and  the  data  for  the 
evaluation  would  be  presentations  and  background  material  prepared  by  the  Program 
Managers  and  Principal  Investigators.  The  panel  evaluation  could  follow  its  current 
open  meeting  consensus-building  approach,  or  use  a  Delphi-type  consensus-building 
method  such  as  RAND  s  E-DEL+I  that  was  used  in  this  project.  RAND  would  then 
compile  the  data  and  develop  and  analyze  the  portfolio  charts. 

Another  possible  implementation  of  this  decision  framework  might  be  to  evalu¬ 
ate  specific  ONR  project  portfolios  on  a  periodic  basis  by  a  standing  expert  panel, 
(e.g.,  appointed  to  staggered  three-year  terms  with  one-third  of  the  panel  new  each 
year).  The  RAND  E-DEL+I  consensus  building  method  might  be  used  to  structure 
the  panels  deliberations  together  with  the  RAND  PortMan  decision  framework  used 
to  perform  a  portfolio  analysis  and  develop  an  investment  strategy,  in  a  fashion  similar 
to  that  described  in  Chapters  4  and  5. 
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This  application  of  the  decision  framework  could  be  accomplished  with  a  first 
round  by  email,  and  the  additional  rounds  during  a  one-day  or  two-day  meeting  at 
which  the  data  on  capability,  performance  potential,  and  transition  probability  would 
be  developed  and  plotted  on  the  portfolio  chart.  The  panel  could  then  debate  both 
the  position  of  the  projects  on  the  portfolio  chart  and  the  likely  direction  of  motion. 
Feedback  could  be  provided  to  the  ONR  Program  Managers  and  Principal  Investiga¬ 
tors  concerning  where  they  were  placed  on  the  chart  and  why. 

This  standing  panel  meeting  could  be  used  as  a  tool  to  redirect  funding  towards 
the  most  promising  projects,  with  the  data  and  analysis  updated  at  each  meeting.  A 
note  of  caution:  successful  R&D  projects  take  time,  so  one  would  not  expect  to  see 
wholesale  changes  in  funding.  Perhaps  projects  might  be  reviewed  on  a  yearly  basis, 
and  funding  changes  made  only  after  multiple  unsatisfactory  reviews.  Such  yearly 
reviews  could  enable  ONR  management  to  emphasize  higher  payoff  R&D,  and  to 
track  progress  towards  achieving  needed  performance,  as  well  as  responding  to  chang¬ 
ing  requirements  in  a  timely  fashion,  consistent  with  sound  R&D  management  prac¬ 
tices.  Whether  it  is  applied  to  a  subprogram,  program,  or  entire  class  of  projects  (e.g., 
6.2,  applied  research,  or  6.3,  advanced  technology  development),  RANDs  PortMan 
decision  framework  can  be  tailored  both  to  the  scope  of  the  evaluation  and  to  the  avail¬ 
able  resources.  Projects  can  be  evaluated  in  groups  to  achieve  the  desired  number  of 
individual  units  for  the  evaluation.  The  amount  of  data  used  in  the  review  would  then 
be  the  maximum  consistent  with  the  resources  available  for  the  evaluation. 
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CHAPTER  ONE 

Introduction 


This  chapter  briefly  describes  the  responsibilities  and  challenges  facing  the  sponsor  of 
this  study,  The  Office  of  Naval  Research  (ONR).  It  then  briefly  introduces  RANDs 
Portfolio  Management  (PortMan)  decision  framework  and  explains  how  this  approach 
is  aimed  at  meeting  ONRs  challenges.  The  chapter  concludes  with  an  explanation  of 
the  role  of  U.S.  Naval  Reserve  (USNR)  officers  in  supporting  this  study  and  an  outline 
of  this  report. 


ONR  Responsibility  for  Three  Different  Navies 

The  ONR  has  the  responsibility  for  defining  and  sponsoring  research  and  development 
(R&D)  necessary  to  support  the  requirements  of  three  different  Navies  and  Marine 
Corps:  the  current  Navy  and  Marine  Corps,  the  Navy  and  Marine  Corps  of  the  next 
five  years,  and  the  “Navy  and  Marine  Corps  After  Next”  of  perhaps  20  years  into  the 
future  (Saalfeld,  2001,  Gaffney  et  al.,  2000).  Since  the  R&D  requirements  of  each  of 
these  Navies  and  Marine  Corps  are  vastly  different,  ONR  must  fund  a  broad  spectrum 
of  research,  ranging  from  6.1  basic  research,  needed  to  open  up  new  options  for  the 
long-term,  to  very  near-term  6.3  advanced  technology  development  to  support  the 
current  fleet.1  Moreover,  ONR  must  make  its  R&D  funding  decisions  in  the  pres¬ 
ence  of  uncertainty:  uncertainty  in  required  capabilities,  uncertainty  in  performance 
requirements,  and  uncertainty  in  the  feasibility  of  a  technology  or  R&D  approach. 


1  Appendix  A  lists  the  current  DoD  definitions  of  the  categories  of  R&D.  ONRs  responsibility  is  to  sponsor  6.1- 
6.3,  basic  and  applied  research  and  advanced  technology  development,  and  to  transition  it  to  users  and  programs 
that  sponsor  and  conduct  6.4  and  higher  levels  of  R&D  leading  to  implementation.  ONR  is  uniquely  charged  with 
responsibility  for  both  Discovery  and  Invention  (6.1  and  most  of  6.2)  and  Exploration  and  Development  (the  most 
mature  part  of  6.2  and  all  of  6.3).  The  spectrum  of  6. 1-6.3  R&D  is  defined  as  science  and  technology  (S&T). 
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The  Challenge  of  Use-Inspired  Basic  Research 

To  successfully  execute  its  mission,  ONR  must  pursue  cutting-edge  basic  research  in 
scientific  areas  that  are  of  unique  interest  to  the  Navy,  build  on  the  basic  research 
advances  made  by  ONR  and  other  organizations  in  areas  of  joint  interest,  and  direct 
its  applied  research  and  technology  development  toward  user  requirements  that  will 
facilitate  transition.  Its  basic  research  component  thus  cannot  be  only  curiosity-driven, 
but  must  also  be  strongly  driven  by  a  sense  of  the  potential  use  of  new  technological 
options  and  capabilities  by  the  Navy  and  Marine  Corps  After  Next.  This  use-inspired 
basic  research  combines  the  most  rigorous  approach  to  fundamental  understanding 
with  a  strong  focus  on  solving  real-world  problems,  much  in  the  spirit  of  Louis  Pas¬ 
teur  s  groundbreaking  R&D  successes  in  biology  and  medicine  (Stokes,  1997). 


The  Challenge  of  Facilitating  Transition 

ONRs  challenge  of  facilitating  transition  is  similar  to  that  faced  by  commercial  firms 
who  often  find  that  the  exponential  increase  in  funds  required  for  product  development, 
coupled  with  the  lag  in  cash  flow  from  sales,  leads  to  a  “valley  of  death”  for  many  poten¬ 
tial  products  that  emerge  from  successful  R&D  programs.  To  overcome  this  challenge, 
ONR  several  years  ago  instituted  the  Future  Naval  Capabilities  (FNC)  program.  FNCs 
have  specific  6.3  objectives  for  enabling  capabilities  defined  by  Integrated  Product  Teams 
(IPTs),  which  include  representatives  from  the  science  and  technology  (S&T),  require¬ 
ments,  and  acquisition  communities.  Contingent  on  meeting  these  6.3  objectives,  “spike” 
funding  is  committed,  including  6.4  funds,  to  accomplish  the  transition. 


Closing  the  Basic  Research-Technology  Development  Gap 
with  6.2  Research 

ONR  s  R&D  investment  strategy  must  balance  two  sometimes  competing  roles:  align¬ 
ing  priorities  with  user  requirements  to  accomplish  transition,  while  continuing  to 
nurture  and  develop  the  S&T  base  that  comes  from  use-inspired  basic  research.  In  this 
report,  we  will  focus  on  the  middle  portion  of  ONRs  R&D  spectrum  (6.2),  for  it  is 
here  that  the  “rubber  meets  the  road”  in  linking  up  the  basic  research  advances  of  ONR 
and  others  with  the  Navy  and  Marine  Corps  user  requirements  that  must  be  met  in 
order  to  successfully  transition  technologies. 

The  Chief  of  Naval  Research,  RADM  Jay  M.  Cohen,  recently  described  ONRs 
basic  research  investment  strategy  as  “planting  a  thousand  flowers,  to  get  100  projects, 
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Figure  1.1 

Schematic  View  of  the  Transition  from  Basic  Research  Investment  to  Meeting  Navy  and  Marine 
Corps  Requirements 
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three  prototypes,  and  one  profit-maker.”2  Our  objective  in  this  report  is  to  describe 
an  approach  and  a  quantitative  method  for  evaluating  those  flowers  and  projects,  as 
they  emerge,  in  order  to  develop  investment  strategies  that  can  nurture  and  grow  the 
most  promising  projects  among  them.  This  method  must,  and  will,  allow  for  periodic 
reexamination  and  reevaluation,  as  conditions  change  and  the  technologies  mature,  to 
sharpen  the  investment  strategy  and  guide  the  projects  to  a  successful  transition  that 
meets  user  requirements. 

Figure  1 . 1  schematically  illustrates  the  process  via  which  advances  resulting  from 
basic  research  investments  are  built  upon  through  6.2  and  6.3  research  to  meet  Navy 
and  Marine  Corps  requirements.  At  the  intermediate  6.2  stage  the  “market  pull” 
requirements  must  be  matched  by  building  on  the  “technology  push”  from  success¬ 
ful  6.1  research  results.  Our  principal  objective  in  this  work  is  to  provide  a  means  for 
evaluating  R&D  investments  in  terms  of  their  ability  to  facilitate  this  matching  and 
thus  close  the  gap  between  6.1  research  and  Navy  and  Marine  Corps  requirements. 

2  Interview  quoted  in  Sea  Power,  February  2004,  p.  29.  A  detailed  study  of  project  literature,  patent  literature  and 
experience,  and  venture  capital  experience  (Stevens  and  Burley,  1997)  validates  this  approach  with  industrial  experi¬ 
ence,  suggesting  that  it  takes  3000  new  ideas  to  achieve  one  commercial  success. 
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Figure  1.2 

Schematic  View  of  the  Navy  R&D  Process 


Figure  1.2  schematically  illustrates  the  process  of  matching  the  capabilities  to  be 
developed  through  R&D  to  those  that  meet  end  user  requirements.  To  define  these 
capabilities,  we  take  a  warfighting  scenario  approach.  The  benefit  of  such  an  approach 
is  that  one  can  readily  identify  which  capabilities  are  useful  to  the  scenario  objectives, 
and  even  under  some  circumstances  quantify  performance  levels  required  to  achieve 
them.  For  example  a  recent  RAND  report  (Perry  et  al.,  2002)  describes  an  island 
nation  defense  scenario  and  related  measures  of  effectiveness  for  technological  perfor¬ 
mance  based  on  achieving  the  scenario  objectives.3 

The  process  for  which  we  wish  to  develop  a  sound  investment  strategy  is  the 
matching  of  the  performance  of  technologies  that  emerge  from  6. 1  research  to  the  per¬ 
formance  needed  to  meet  the  requirements  for  capabilities  that  will  perform  in  Navy 
and  Marine  Corps  mission-critical  scenarios.  We  note  that  the  6.1  base  on  which  the 
R&D  advances  are  built  is  the  combined  result  of  research  sponsored  by  ONR  and 
other  agencies.  In  certain  cases,  these  results  are  already  a  close  enough  match  that  they 
can  directly  transition  (e.g.,  signal  processing  algorithms  that  may  be  directly  adopted 
by  industry).  On  the  other  side  of  the  spectrum  are  6.3  projects  that  work  with  already 
mature  capabilities  to  facilitate  the  transition  to  6.4  and  higher  levels  of  R&D.  The 


3  We  adopted  this  scenario  as  a  concrete  example  within  which  to  initially  evaluate  capabilities  proposed  for  devel¬ 
opment  through  R&D  for  the  case  study.  The  downside  of  this  approach  is  that  there  are  many  more  warfighting 
scenarios  that  may  be  equally  as  important  as  or  even  more  important  than  this  island  scenario.  As  described  below 
in  the  methods  chapter,  we  sought  to  balance  the  scenario  approach  by  including  two  components  in  the  estimate  of 
capability  value,  the  first  dependent  upon  usefulness  in  the  island  scenario,  and  the  second  dependent  upon  breadth 
of  usefulness  in  a  range  of  scenarios  identified  as  important  by  the  Navy. 
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focus  of  this  report  is  on  those  capabilities  that  are  not  yet  mature ,  and for  which  6,2  projects 
may  improve  the  match  sufficiently  to  reach  a  level  of  attractiveness  to  the  end  users  that 
can  effectively  bridge  the  “valley  of  death”  alluded  to  previously. 


The  RAND  PortMan  R&D  Portfolio  Management  Decision  Framework 

This  section  describes  RANDs  PortMan  R&D  decision  framework.  We  note  that  the 
purpose  of  PortMan  is  to  evaluate  a  defined  group  of  actual  or  proposed  projects  and  to 
provide  a  means  for  creating  a  portfolio  from  them  that  maximizes  the  value  of  R&D 
investments.  It  does  not  generate  an  absolute  score  for  the  total  portfolio  that  could 
be  used  to  compare  to  portfolios  of  other  projects  or  to  proportionally  allocate  funds 
between  portfolios  of  different  projects. 

RAND’s  PortMan  R&D  decision  framework  (Silberglitt  and  Sherry,  2002) 
approaches  the  R&D  investment  strategy  problem  by  evaluating  the  expected  value 
(EV)  of  each  R&D  project  based  upon  the  answers  to  three  fundamental  questions: 

1.  What  is  the  value  to  the  end-users  of  the  capability,  at  a  specified  performance 
level,  that  the  R&D  is  aimed  at  achieving? 

2.  What  is  the  potential  of  the  R&D,  if  successful,  to  achieve  the  performance  level 
required  for  this  capability? 

3.  What  is  the  probability  that  the  R&D  approach  will  be  successful  and  will  be 
transitioned  and  fielded? 

The  EV  is  computed  as  the  product  of  capability,  performance  potential,  and  transition 
probability.4  Thus  performance  potential  and  transition  probability  become  scale  factors 
that  reduce  the  value  of  the  capability  that  would  be  achieved  with  full  performance  and 
project  transition  and  fielding.  It  is  this  scaling  of  value,  plus  the  portfolio  analysis 
methods  described  in  Chapters  3  and  5,  which  define  the  PortMan  approach.  PortMan 
can  be  applied  with  scenario-based  capability  values,  as  in  the  case  study  described  in 
Chapters  4  and  5,  or  any  alternative  valuation  method.5 

The  answers  to  the  above  questions  depend  on  experts’  estimates  of:  (1)  the  value 
to  the  Navy  and  Marine  Corps  of  meeting  specific  requirements;  (2)  the  potential  for 
achieving  specific  levels  of  performance;  and  (3)  the  quality  of  transition  planning  and 
activities.  All  of  these  quantities  may  change  over  time.  Effective  use  of  this  framework 


4  The  equation  for  Expected  Value  used  in  PortMan  is  based  on  an  adaptation  of  Decision  Theory  (see,  e.g.,  Raiffa, 
1997)  as  described  in  detail  in  Silberglitt  and  Sherry  (2002). 

5  A  notable  approach  for  value  estimation  is  described  in  Saunders  et  al.,  1995,  which  uses  2-D  plots  of  relative 
value  versus  breadth  of  demand  to  determine  the  value  to  the  Navy  of  specific  capabilities.  PortMan  provides  a  way 
of  evaluating  the  extent  to  which  R&D  on  specific  projects  might  achieve  the  level  of  performance  necessary  to 
achieve  these  capabilities. 


6  Portfolio  Analysis  and  Management  for  Naval  Research  and  Development 


depends  on:  collecting  the  best  available  data  on  requirements,  performance  potential, 
and  transition  planning  and  activities  for  each  project;  integrating  the  judgments  of 
the  best  available  experts  based  on  these  data;  and  repeating  the  process  at  reasonable 
intervals  to  support  a  dynamic  investment  strategy  that  takes  advantage  of  emerging 
opportunities. 

The  use  of  PortMan  to  analyze  a  project  portfolio  and  develop  an  investment 
strategy,  as  described  in  Chapters  3  and  5,  draws  on  a  number  of  approaches  that  have 
been  described  in  the  industrial  research  management  literature  (see,  e.g.,  Cooper  et 
al.,  1997).  As  noted  by  Cooper  et  ah,  portfolio  management  has  three  broad  goals: 
value  maximization;  balance;  and  strategic  direction.  PortMan  maximizes  the  value 
of  the  portfolio  by  computing  and  analyzing  the  components  of  EV  for  each  project 
(see,  e.g.,  Figures  5.1  through  5.7).  It  allows  explicit  comparisons  of  project  charac¬ 
teristics  that  allow  the  manager  to  balance  the  portfolio  (e.g.,  as  concerns  risk  versus 
reward).  Finally,  it  allows  development  of  an  investment  strategy  for  this  portfolio  that 
is  explicitly  aimed  at  achieving  capabilities  that  reflect  the  strategic  needs  identified  in 
the  scenarios  (or  other  means)  used  to  define  warfighter  value. 

Unique  Aspects  of  the  RAND  Decision  Framework 

We  note  here  two  principal  differences  between  this  approach  and  the  conventional 
evaluation  of  R&D  projects:  (1)  the  EV  is  a  product,  not  a  weighted  sum  of  factors; 
and  (2)  there  is  an  uncertainty  range,  not  a  point  solution.  Elaborating  on  point  (2),  we 
do  not  propose  to  use  the  EV  as  the  sole  benchmark  for  comparing  projects.  Rather,  we 
will  evaluate  and  compare  R&D  projects  based  upon  the  three  components  of  the  EV, 
including  uncertainty,  plus  their  estimated  direction  of  change  over  time.  The  invest¬ 
ment  strategy  question  then  will  be:  How  will  my  investment  of  $X  in  this  project 
cause  these  quantities  to  change  and  how  much  is  that  change  worth?  Returning  to  the 
flower  analogy  of  the  previous  section,  we  want  to  provide  the  nurturing  and  support 
that  will  allow  the  most  promising  buds  to  bloom  and  prosper.  As  discussed  in  Chap¬ 
ters  3  and  5,  the  analysis  provided  by  using  this  framework  will  allow  the  development 
of  investment  strategies  tailored  to  the  strengths  and  weaknesses  of  specific  research 
projects. 

There  is  a  strong  similarity  here  to  recent  work  in  the  financial  options  arena 
(Luehrman,  1998)  in  which  options  are  viewed  as  investments  in  a  tomato  garden. 
You  never  want  to  invest  in  rotten  tomatoes  (low  on  all  components  of  EV)  and  you 
always  want  to  invest  now  in  ripe  tomatoes  (high  on  all  components  of  EV).  The  chal¬ 
lenge  is  to  figure  out  how  and  when  to  invest  your  limited  resources  on  the  tomatoes 
in  between  (i.e.,  how  to  pick  the  options  that  are  developing  well  enough  so  that  with 
your  help  they  become  more  likely  to  mature  in  your  required  time  frame).  Likewise, 
our  problem  is  to  identify  the  R&D  projects  that  have  sufficient  promise  that  with 
investment,  they  may  mature  to  meet  requirements  and  be  transitioned.  This  requires 
evaluation  based  on  best  current  information,  and  tracking  over  time  as  conditions 
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change  and  the  technology  matures.  Some  of  these  projects  will  be  of  high  value  to 
the  warfighter,  but  with  formidable  problems  remaining  to  be  solved.  The  question  for 
these  is:  Can  our  investment  move  them  into  the  promising  and  now  ripening  region 
of  the  tomato  garden? 

Using  Expert  Judgment  to  Estimate  the  EV  Components 

This  case  study  application  of  RANDs  PortMan  decision  framework  used  RAND’s 
E-DEL+I™  collaborative  consensus  building  method  (Wong,  2003)  with  an  expert 
panel  of  Naval  officers  and  senior  civilian  executives.  This  method  retains  the  inde¬ 
pendent  assessment  features  of  the  traditional  Delphi  method  (Dalkey,  1969),  with 
each  expert  evaluating  the  capability,  performance  potential,  and  transition  probability 
of  the  same  20  sample  ONR  applied  research  projects  using  the  same  data,  and  then 
reevaluating  after  receiving  composite  anonymous  feedback  showing  how  the  other 
members  of  the  panel  evaluated  the  same  projects.  After  these  traditional  first  two 
rounds  of  Delphi  evaluation  are  completed,  E-DEL+I  uses  a  teleconference  between 
the  participants  to  provide  verbal  feedback  in  addition  to  the  traditional  written  feed¬ 
back.  Participants  are  not  identified  during  the  teleconference,  which  is  facilitated  by 
an  independent  expert  and  focuses  on  discussion  of  the  questions  on  which  the  par¬ 
ticipants  had  the  largest  variation  of  responses  in  the  first  two  rounds.  In  this  way, 
E-DEL+I  allows  for  a  mix  of  written  and  verbal  arguments,  thus  balancing  between 
participants  who  are  better  at  one  medium  or  the  other.  The  third  evaluation  round  is 
performed  during  a  break  in  the  teleconference  and  the  fourth  after  the  teleconference 
is  completed.  Both  of  these  rounds  use  the  same  independent  and  anonymous  evalua¬ 
tion  as  the  first  two  rounds,  thus  preserving  one  of  the  hallmarks  of  the  traditional  Del¬ 
phi  method.  Finally,  E-DEL+I  has  proven  to  be  highly  timely  and  cost-effective,  since 
it  is  conducted  via  e-mail  and  teleconference,  and  hence  does  not  require  any  travel. 
For  example,  participants  in  the  exercise  described  herein  were  located  in  California, 
Rhode  Island,  Virginia,  and  Washington,  D.C. 


Role  of  USNR  on  the  Project  Team 

RAND  was  supported  in  the  work  described  in  this  report  by  officers  of  the  USNR 
Program  38,  assigned  to  ONR.  The  role  of  the  USNR  was  principally  in  two  areas: 
(1)  definition  of  value  to  the  warfighter;  and  (2)  development  of  a  plan  for  integrating 
the  RAND  PortMan  decision  framework  into  the  overall  Navy  R&D  decision-mak¬ 
ing  process.  In  particular,  USNR  officers  developed  background  information  on  the 
Navy’s  approach  to  S&T  metrics,  prepared  the  scenario  description  that  was  used  by 
the  expert  panel  in  the  Delphi  evaluation  of  ONR  applied  research  projects,  assisted  in 
the  formation  of  the  expert  panel,  and  participated  throughout  the  work  in  discussions 
between  RAND  and  ONR  staff. 
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Outline  of  the  Report 

Hie  balance  of  this  report  is  organized  as  follows:  Chapter  2  provides  background 
and  information  on  current  approaches  within  the  Navy  for  R&D  project  evaluation 
and  provides  a  context  for  the  RAND  PortMan  decision  framework  in  terms  of  other 
relevant  decision-making  aids.  Chapter  3  summarizes  the  salient  features  of  the  two 
RAND  methods  used  in  this  work — the  RAND  PortMan  decision  framework,  and  the 
RAND  E-DEL+I  consensus  building  method.  Chapter  4  provides  the  details  of  the 
case  study  evaluation  of  20  ONR  Code  31  (Information,  Electronics,  and  Surveillance 
Department)  6.2  R&D  projects.  Chapter  5  analyzes  the  results  of  the  case  study  and 
discusses  how  to  use  such  results  to  perform  portfolio  analysis  and  develop  an  invest¬ 
ment  strategy.  Chapter  6  describes  RAND’s  conclusions  based  upon  the  case  study 
results  and  RAND’s  recommendations  for  the  use  of  the  decision  framework  for  R&D 
portfolio  analysis  and  investment  strategy  development.  Five  appendices  provide  addi¬ 
tional  details  relevant  to  the  material  described  in  the  text  of  the  report.  Appendix  A 
lists  the  official  Department  of  Defense  definitions  of  the  stages  of  research  discussed 
throughout  this  report.  Appendices  B,  C,  D,  and  E  provide  background  materials  used 
in  the  case  study  evaluation. 


CHAPTER  TWO 

Background 


The  portfolio  management  decision  framework  described  in  this  report  must  operate 
within  the  context  of  existing  Navy  institutions  and  procedures.  Prior  to  the  initiation 
of  RANDs  work  on  adapting  its  framework  for  ONR,  CAPT  Aaron  Watts  (USNR) 
carried  out  an  informal  survey  of  how  Navy  and  Department  of  Defense  (DoD)  orga¬ 
nizations  involved  in  R&D  currently  measure  or  evaluate  their  S&T  investments.  He 
found  a  highly  eclectic  mix  of  approaches,  ranging  from  oral  boards  based  on  detailed 
written  submissions,  to  computer-based  tools  that  use  specific  anchored  scales.  Hie 
definition  of  measures  of  effectiveness  and  the  data  collection  requirements  to  esti¬ 
mate  these  measures  differed  substantially  between  organizations,  so  comparison  of  the 
RAND  approach  to  a  “generic”  Navy  approach  was  not  possible.  However,  two  specific 
recent  examples  of  Navy  S&T  evaluations  provide  a  context  for  the  present  work. 


Department  of  the  Navy  6.3  Review 

The  Department  of  the  Navy  (DoN)  recently  performed  an  internal  review  of  its  entire 
Fiscal  Year  2000  Advanced  Technology  Development  (6.3)  program  (Kostoff,  Miller, 
andTshiteya,  2001).  A  31-member  panel,  composed  of  senior  Naval  Officers  and  civil¬ 
ian  executives  drawn  from  ONR,  The  Marine  Corps,  the  DoN  S&T  resource  spon¬ 
sor  (Office  of  the  Chief  of  Naval  Operations,  OPNAV  911),  and  operational  Navy 
organizations  responsible  for  setting  requirements  performed  this  review.  The  review 
panel  was  provided  with  data  prepared  and  presented  by  55  different  programs,  based 
on  the  taxonomy  by  which  programs  were  selected  and  managed.  The  data  were  tar¬ 
geted  toward  a  specific  set  of  evaluation  criteria:  Military  Goal  (importance  to  each 
of  12  designated  FNCs),  Military  Impact  (potential  for  military  capability  improve¬ 
ment),  Technical  Approach  (technical  payoff  versus  alternatives),  Program  Executabil- 
ity  (probability  of  demonstration),  and  Transitionability  (probability  of  transition). 
Descriptions  were  provided  for  rankings  of  high,  medium,  and  low  for  each  of  these 
criteria,  as  well  as  for  an  overall  item  evaluation  (based  on  incremental  versus  revolu- 
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tionary  improvement  in  military  and  technology  capabilities).  In  addition  to  the  6.3 
presentations,  the  panelists  received  situation  report  presentations  from  the  Chairs  of 
each  of  the  FNC  IPTs. 

While  the  scope  of  this  evaluation  (55  programs  averaging  approximately  $10  mil¬ 
lion  each)  was  far  greater  than  that  of  our  20-project  case  study,  there  are  some  important 
similarities  in  approach.  Both  used  a  panel  of  senior  Naval  officers  and  civilian  execu¬ 
tives  drawn  from  organizations  with  diverse  perspectives.  Both  provided  the  panel  with 
data  targeted  toward  specific  evaluation  criteria  based  on  importance  of  the  capability  to 
the  Navy,  potential  to  achieve  improved  capability,  and  probability  of  transitioning  that 
capability  to  the  Fleet.  Both  also  provided  value  descriptions  for  their  ranking  scales. 

However,  there  are  some  clear  differences  between  the  DoN  6.3  review  and  the 
application  of  the  RAND  PortMan  decision  framework  to  evaluate  ONR  applied 
research  (6.2)  projects.  The  DoN  6.3  review  asked  each  panelist  to  assign  a  “bottom- 
line  quality  score”  (Overall  Item  Evaluation).  As  briefly  summarized  in  Chapter  1  and 
described  in  more  detail  in  Chapters  3  and  5,  RAND  computes  an  EV  as  a  product 
of  the  capability,  performance  potential,  and  transition  probability  scores  of  each  pan¬ 
elist  and  accounts  for  uncertainty,  not  using  a  point  solution.  Furthermore,  RAND 
performs  a  portfolio  analysis  using  the  range  of  capability,  performance  potential,  and 
transition  probability  values  assigned  by  the  panelists.  This  reflects  the  differences  in 
potential  payoff  and  maturity  of  the  projects,  and  allows  comparative  analysis  of  the 
possible  leverage  of  R&D  investment  and  management  attention  for  individual  proj¬ 
ects,  or  groups  of  projects.  Another  difference  of  less  significance  is  that  RAND  staff 
compiled  the  data  for  the  evaluation  based  on  proposal  information  provided  by  ONR. 
This  was  dictated  by  time  and  resource  constraints.  The  RAND  PortMan  framework 
could  certainly  be  used  with  data  prepared  and/ or  presented  by  Naval  staff. 


The  ONR  Code  31  Program  Evaluation  Process 

The  following  description  of  the  program  evaluation  process  currendy  used  by  ONR 
Code  31  (Information,  Electronics,  and  Surveillance  Department)  is  a  summary  of  an 
evaluation  report  written  by  CDR  Vance  Brahosky  (USNR). 

ONR  Code  31  uses  a  program  evaluation  model  based  on  a  “Board  of  Directors” 
template.  This  model  makes  use  of  leaders  drawn  from  the  various  research  laborato¬ 
ries,  acquisition  program  offices,  Systems  Commands,  and  OPNAV  who  meet,  on  an 
annual  basis,  in  a  technology  review  known  formally  as  the  “ONR  6.2  Review”  and 
informally  as  “the  Gathering.”  During  the  course  of  this  Gathering,  current  fiscal  year 
Discovery  and  Invention  (D&I)  6.2  programs  are  reviewed  and  next  fiscal  year  D&I 
6.2  proposals  (new  starts)  are  evaluated. 

Dr.  Bobby  Junker,  Head  of  Code  3 1 ,  chairs  the  reviews  conducted  for  each  of  the 
functional  areas  under  his  cognizance  (Navigation,  Electronic  Warfare,  Command  and 
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Control,  Electronics  Technology,  Surveillance  and  Communications).  This  is  a  phased 
review  that  dedicates  separate  time  periods  to  focus  on  each  of  the  six  functional  areas. 
Each  area  review  is  typically  allotted  between  3  and  5  days,  with  a  review  panel  size 
between  about  8  and  about  18  individuals,  depending  on  the  scope  of  the  program. 
Dr.  Junker  provides  a  framework  for  the  decisionmaking  process  in  the  form  of  guid¬ 
ance  that  summarizes  current  Navy  visions  and  strategy,  identifies  Code  31  priority 
technical  areas  and  capabilities,  and  details  the  requirements  for  technical  presenta¬ 
tions.  The  basic  data  for  the  evaluation  are  presentations  of  the  ongoing  6.2  programs 
and  the  proposed  new  6.2  starts,  which  are  made  by  the  relevant  Program  Managers 
and  Principal  Investigators.  To  provide  a  perspective  for  the  6.2  review,  relevant  por¬ 
tions  of  the  basic  research  (6.1)  program  and  of  the  FNC  programs  are  also  presented 
to  the  panel. 

The  Code  31  program  evaluation  process  uses  a  prescreening  of  solicited  white 
papers  from  laboratories  and  warfare  centers  to  select  those  new  start  proposals  to 
be  submitted  to  the  panel  review  at  the  Gathering.  This  prescreening  process  uses 
a  three-person  panel  of  ONR  program  managers  (which  we  call  the  “prescreening 
panel”  to  differentiate  it  from  the  review  panel  at  the  Gathering)  to  evaluate  the  white 
papers.  The  prescreening  panel  members  evaluate  each  white  paper  on  subjective  1-10 
scales  for  each  of  two  criteria:  “S&T  Technology  Value”  and  “Transition  Potential. 
The  scores  on  each  of  these  criteria  are  added  together  to  get  a  total  score,  and  each 
member  of  the  prescreening  panel  provides  a  priority  order  based  on  their  total  scores 
for  each  white  paper.  The  prescreening  panel  then  meets  as  a  group  and  puts  together  a 
consensus  priority  list.  The  new  start  proposals  that  are  presented  at  the  Gathering  are 
based  on  the  white  papers  that  came  out  on  top  on  the  prescreening  panel  consensus 
priority  list.  The  number  varies  between  programs,  but  is  typically  larger  than  can  be 
funded  with  the  projected  fiscal  year  new  start  budget. 

During  the  Gathering,  the  review  panel  scores  each  new  start  proposal,  as  it  is 
briefed,  and  turns  in  a  completed  score  sheet  at  the  end  of  each  briefing.  The  evaluation 
criteria  are:  (1)  Technical  Rationale  (i.e.,  will  success  of  this  project  have  a  significant 
impact  on  the  scientific  and  technical  state-of-the-art),  (2)  Department  of  the  Navy 
Rationale  (i.e.,  what  DoN  critical  need  is  addressed  by  this  effort),  and  (3)  Program¬ 
matic  Rationale  (i.e.,  why  is  now  the  right  time  to  do  this  work).  These  score  sheets  are 
compiled  and  used  as  the  basis  for  a  final  panel  review  meeting  that  is  held  at  the  con¬ 
clusion  of  the  Gathering.  At  this  final  review  meeting,  the  panel  develops  and  submits 
to  Code  31a  consensus  list  of  priority  recommendations  for  new  starts. 

Like  the  DoN  6.3  Review,  the  ONR  Code  31  6.2  evaluation  process  bears  some 
similarities  to  the  RAND  PortMan  decision  framework.  It  uses  a  broadly  based  expert 
panel  to  perform  the  evaluation  and  bases  the  evaluation  on  data  that  are  targeted 
toward  specific  evaluation  criteria  related  to  value  of  capabilities  to  the  Navy,  techni¬ 
cal  performance,  and  transition  probability.  However,  there  are  also  some  important 
differences.  While  the  Code  31  process  uses  specific  evaluation  criteria,  it  bases  its 
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evaluations  on  subjective  scores,  not  anchored  scales  with  specific  value  descriptions. 
Also,  like  the  DoN  6.3  review,  the  Code  31  process  uses  a  point  solution, (i.e.,  a  single 
numerical  priority  ranking  of  projects),  whereas,  as  noted  above,  RAND  computes  an 
EV  and  performs  a  portfolio  analysis,  including  estimates  of  uncertainty  in  capabil¬ 
ity,  performance  potential,  and  transition  probability.  Finally,  the  Code  31  process 
uses  presentations  to  provide  the  evaluation  data,  whereas  in  the  case  study  evaluation 
RAND  compiled  project  summaries  based  on  the  proposal  data.  As  noted  above,  this 
was  dictated  by  time  and  resource  constraints.  The  RAND  PortMan  framework  could 
certainly  be  used  with  data  prepared  and/or  presented  by  Program  Managers  and  Prin¬ 
cipal  Investigators. 


Related  R&D  Evaluation  Approaches 

Two  key  features  of  this  RAND  approach  are  its  use  of  anchored  scales  (i.e.,  scales  that 
include  explicit  descriptions  of  the  requirements  or  thresholds  for  assigning  particular 
values)  and  its  recognition  of  uncertainty  in  the  estimation  of  the  components  of  the 
EV:  capability,  performance  potential,  and  transition  probability. 

Anchored  scales  have  recently  been  used  by  the  Industrial  Research  Institute  (IRI) 
(Davis  et  ah,  2001)  to  develop  a  method  for  estimating  the  probability  of  success  of 
commercial  R&D.  Technical  factors  were  proprietary  position,  competencies,  com¬ 
plexity,  access  to  technology,  and  manufacturing  capability.  Commercial  factors  were 
market  need,  brand  recognition,  distribution  channels,  customer  strength,  raw  materi¬ 
als,  and  safety,  health,  and  environmental  risks.  For  each  factor,  the  authors  wrote  a 
description  characterizing  rankings  of  1  through  5.  These  scales  were  then  used  by  an 
IRI  committee  to  evaluate  a  specific  business  opportunity.1  The  use  of  the  anchored 
scales  led  to  explicit  recognition  of  not  only  the  pros  and  cons  of  the  opportunity,  hut  also 
the  reasons  for  them.  This  allowed  a  dialog  between  committee  members  on  whether  or 
not  to  pursue  the  opportunity  now,  and  how  to  improve  its  chances  for  success  in  the 
future. 

The  RAND  PortMan  decision  framework  uses  anchored  scales  in  a  similar  fashion. 
By  requiring  the  evaluators  to  answer  specific  questions  concerning  capability,  perfor¬ 
mance  potential,  and  transition  probability,  this  RAND  framework  collects  and  records 
the  information  needed  to  analyze  the  positive  and  negative  aspects  of  each  R&D  proj¬ 
ect,  and  to  facilitate  discussion  and  analysis  of  possible  investment  strategies. 

R&D  is  by  its  nature  performed  in  the  face  of  uncertainty.  The  reason  for  pursu¬ 
ing  the  R&D  effort  is  to  increase  knowledge  and  capability  in  a  way  that  enables  new 
approaches,  processes,  and  technologies.  But  there  is  always  the  risk  that  the  R&D  will 
fail  to  meet  its  objectives.  R&D  investment  strategies  thus  attempt  to  balance  this  risk 


1  This  was  a  realistic  example  constructed  for  the  analysis,  not  an  actual  opportunity. 
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against  potential  payoff.  Because  it  is  possible  to  cancel  unsuccessful  projects  before 
they  reach  the  most  expensive  (product  development)  stage,  it  has  in  fact  been  argued 
based  on  analysis  of  the  earned  value  of  R&D  that  projects  with  higher  risk  and  higher 
payoff  are  preferred  (Morris,  Teisberg,  and  Kolbe,  1991). 

The  R&D  investment  decisions  we  are  considering  in  this  report  are  not  one-time 
actions,  but  rather  steps  along  a  particular  technology  or  capability  development  path. 
At  each  stage  along  this  path,  the  investment  decision  will  be  reviewed  and  revisited, 
as  more  information  becomes  available,  both  through  the  R&D  results  and  through 
external  events,  including  the  R&D  and  technology  development  of  others.  In  this 
sense  the  EV  we  compute  using  the  RAND  PortMan  decision  framework  is  really  a 
prospective  value,  because  it  is  based  on  best  estimates,  using  current  data,  of  future 
capability,  performance  potential,  and  transition  probability.  Thus  the  decision  to  fund 
an  R&D  project  can  be  viewed  as  a  decision  to  invest  in  an  option — to  perform  the 
R&D  and  thus  gain  the  knowledge  to  make  a  more  informed  decision  at  a  later  date.2 
In  fact,  there  is  precedent  in  the  commercial  sector  for  the  valuation  of  R&D  invest¬ 
ments  using  financial  options  approaches  (Faulkner,  1996). 

As  described  by  Luehrman,  1998,  one  can  characterize  investment  opportunities 
in  terms  of  their  potential  value-cost  ratio  (v/c),  and  volatility  (V).  High  v/c  with  low 
V  means  invest  now.  Low  v/c  with  high  V  means  invest  probably  never.  The  difficulty 
comes  when  V  is  large,  but  so  is  v/c.  For  R&D,  these  are  the  potentially  high-payoff, 
but  also  high-risk,  investments.  Luehrman  suggests  following  the  change  over  time 
of  v/c  and  V  on  a  2-D  graph  and  making  those  investments  that  are  moving  into  the 
upper  v/c,  lower  V  quadrant.  For  R&D  investments,  this  means  funding  those  R&D 
projects  aimed  at  high-value  capabilities  but  with  remaining  technical  or  other  prob¬ 
lems,  then  tracking  over  time  how  well  the  problems  are  being  solved  and  whether  or 
not  the  value  of  the  capabilities  has  changed.  This  is  precisely  what  the  RAND  Port- 
Man  decision  framework  does  with  its  portfolio  chart,  as  described  briefly  in  Chapter 
1  and  in  greater  detail  in  Chapters  3  and  5. 


2  The  authors  are  indebted  to  Steven  Popper  for  elucidating  the  points  made  in  this  paragraph. 


CHAPTER  THREE 

Methods 


This  chapter  describes  the  two  methods  used  in  this  work:  the  RAND  PortMan  R&D 
decision  framework,  adapted  to  provide  an  R&D  investment  strategy  method  for  the 
Navy;  and  the  RAND  E-DEL+I  consensus  building  method,  used  to  perform  a  case 
study  application  of  this  decision  framework  to  20  ONR  Code  31  6.2  projects,  using 
an  expert  panel. 


Overview  of  the  RAND  PortMan  Portfolio  Management  Decision 
Framework 

As  noted  previously,  the  RAND  PortMan  decision  framework  computes  an  EV  for 
each  R&D  project  as  the  product  of  capability,  performance  potential,  and  transition 
probability.  However,  this  scalar  quantity  combines  several  important  aspects  of  each 
project.  In  order  to  analyze  the  R&D  portfolio,  we  recommend  plotting  a  graph  such 
as  the  one  shown  in  Figure  3.1,  which  also  takes  into  account  the  uncertainty  in  the 
evaluations  of  the  EV  components. 

In  Figure  3.1,  the  x-axis  is  the  transition  probability,  the  y-axis  is  the  capability 
times  the  performance  potential,  the  points  represent  the  average  of  the  values  assigned 
by  the  evaluators  for  each  R&D  project,  using  anchored  scales  as  described  below,  and 
the  circles  or  ovals  around  the  points  indicate  the  spread  of  assigned  values  for  each 
project.1  Constant  EV  contours  are  hyperbolae  on  this  graph,  as  indicated  in  the  figure, 
with  increasing  EV  as  both  x  and  y  increase. 

Figure  3.1  provides  a  visualization  of  the  R&D  portfolio  that  facilitates  compari¬ 
sons  between  the  projects  and  identification  of  investment  opportunities  with  maxi¬ 
mum  leverage.  For  example,  Projects  3  and  6  are  clearly  good  investments — highly 
valued  capabilities  with  good  transition  probability.  Project  8  is  very  likely  to  be  tran- 


1  We  note  that  these  values  are  subjective  estimates  of  expert  evaluators,  based  on  best  available  data,  and  tied  to 
explicit  anchored  scales. 
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sitioned,  but  will  provide  a  much  less  important,  more  incremental,  capability.  Project 
1  is  an  example  of  a  highly  valued  capability  that  currently  has  a  very  low  transition 
probability.  The  question  to  ask  here  is  whether  or  not,  with  investment  and  man¬ 
agement  attention,  the  technical  and  fielding  problems  that  led  to  the  low  transition 
probability  can  be  solved  without  giving  up  the  performance  potential  that  led  to  the 
high  value. 

In  other  words,  the  plot  has  identified  Project  1  as  a  potential  opportunity  that 
requires  management  attention  and  has  possible  high  payoff.  If  it  continues  without 
management  attention,  this  project  might  fail  to  develop  and  execute  a  successful  tran¬ 
sition  plan  within  the  necessary  time.  Or,  if  it  is  not  constrained  sufficiently  by  floors 
on  performance  objectives,  it  might  just  seek  out  and  pursue  the  easiest  possible  tran¬ 
sition  opportunity,  providing  an  incremental  performance  improvement  rather  than 
the  more  likely  transformational  one  that  led  to  its  current  position  on  the  plot.  Simi¬ 
lar  analysis  should  be  performed  for  the  other  projects,  comparing  them  and  group¬ 
ing  them  in  terms  of  potential  payoff  and  investment  and  management  leverage.  An 
example  of  such  an  analysis  is  shown  is  Chapter  5  for  the  case  study  evaluation  of  the 
20  sample  ONR  6.2  projects. 

It  is  important  to  note  that  both  the  average  values  and  the  uncertainties  of  the 
R&D  project  portfolio  are  dynamic  quantities  that  may  change  over  time  as  a  result 
of  technical  results  and  programmatic  and  institutional  changes.  The  RAND  decision 
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framework  provides  a  means  for  periodic  reevaluation  of  the  R&D  projects  and  track- 
ing  and  monitoring  of  the  R&D  portfolio  over  time.  The  combination  of  capability, 
performance  potential,  transition  probability,  and  their  variations  with  time  then  pro¬ 
vides  a  multidimensional  science  and  technology  (S&T)  metric.  This  metric  is  defined 
through  a  transparent,  logical,  auditable  process  using  expert  evaluations  based  upon 
defined  data  and  anchored  scales.  It  also  forms  the  basis  for  a  time-dependent  portfolio 
analysis  that  is  actionable,  fulfilling  a  strong  recommendation  that  emerged  from  the 
DoN  Fiscal  Year  2000  6.3  review  (Kostoff  et  al.,  2001,  p.  295).  The  following  text 
describes  the  approach  that  RAND  took  to  evaluate  each  of  the  EV  components,  and 
discusses  the  anchored  scales  developed  for  the  evaluation,  including  their  data  require¬ 
ments. 

Capability 

In  the  prior  application  of  this  framework  for  the  Department  of  Energy  (Silberglitt 
and  Sherry,  2002),  capabilities  sought  through  materials  R&D  were  valued  in  terms 
of  their  potential  for  energy  savings,  which  was  then  converted  to  dollars  based  upon 
the  estimated  value  of  energy  during  the  years  in  which  the  energy  savings  were  antici¬ 
pated.  Such  an  approach  was  not  possible  in  this  project  because  value  to  the  warfighter 
is  not  easily  measured  using  dollars  or  other  conventional  economic  metrics.  Instead, 
we  adopted  a  scenario-based  approach,  basing  the  value  of  capabilities  sought  through 
R&D  on  their  potential  impact  on  accomplishing  missions  in  specific  warfighting  sce¬ 
narios.  As  a  concrete  example,  we  used  an  island  defense  scenario  that  was  developed 
and  used  recently  to  define  measures  of  effectiveness  to  the  military  of  new  technolo¬ 
gies  (Perry  et  al.,  2002).  This  scenario  hypothesizes  a  small  island  nation  (U.S.  ally) 
facing  a  large  hostile  neighboring  nation  (enemy)  determined  to  annex  the  island.  It 
includes  a  defensive  (cruise  and  ballistic  missile  attack)  and  an  offensive  (time-critical 
target  of  opportunity)  component  to  be  considered  separately.  The  situation  is  provided 
as  context  for  coordinated  operations  involving  individual  units  working  together  to 
achieve  the  defensive  and  offensive  objectives.  The  effect  of  network-centric  and/or 
“traditional”  coordination  and  information  sharing  between  units  is  critical  to  the  out¬ 
come  of  the  scenario.  CDR  Geoffrey  Stothard,  USNR,  prepared  a  detailed  description 
of  this  scenario,  and  specifically  identified  high-level  and  enabling  capabilities  required 
to  successfully  achieve  mission  objectives.  This  scenario  description  is  presented  in 
Appendix  B. 

We  defined2  the  value  of  capabilities  sought  through  R&D  based  on  their  impor¬ 
tance  to  the  island  defense  scenario,  and  also  on  the  extent  to  which  they  were  deemed 
useful  in  other  important  scenarios.  This  dual  approach  recognized  the  value  of  capa- 


2  This  paragraph  and  the  following  paragraphs  describe  the  definitions  of  the  Expected  Value  components  in  the 
PortMan  decision  framework.  The  actual  evaluation  of  these  components  requires  values  assigned  according  to  the 
anchored  scales.  This  was  done  in  the  case  study  by  the  expert  panel,  as  described  in  Chapter  4. 
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bilities  with  pervasive  applicability  to  important  warfighting  scenarios,  but  that  might 
not  have  played  as  prominent  a  role  in  the  island  defense  scenario. 

Performance  Potential 

Each  capability  required  by  the  island  defense  scenario,  or  any  other  scenario,  has 
a  minimum  specified  level  of  performance  needed  to  accomplish  the  mission.  We 
defined  the  performance  potential  of  each  R&D  project  in  terms  of  the  quality  of  the 
match  between  the  technical  performance  that  would  be  achieved  if  the  R&D  were 
successful  and  this  level  of  needed  operational  performance.  Further,  to  achieve  a  high 
value  of  performance  potential,  the  level  of  needed  performance  must  be  accomplished 
under  conditions  that  are  relevant  to  the  scenario  for  which  the  capability  is  required. 
For  example,  consider  a  new  communications  technology  that  increases  speed  and 
bandwidth  of  voice  and  data  transmission.  Its  performance  potential  will  depend  on 
the  extent  to  which  these  increases  actually  enable  the  success  of  coordinated  opera¬ 
tions  in  the  island  defense  scenario  or  other  important  scenarios.  This  will  require  not 
only  improved  technology,  but  also  improved  operation  within  the  constraints  of  the 
scenario  (e.g.,  atmospheric  interference,  specific  sea  states,  lack  of  technical  support). 

Transition  Probability 

Transition  probability  is  the  probability  that  the  capability  will  both  be  achieved  at  its 
potential  level  of  performance,  and  transitioned  to  higher  levels  of  R&D  and  eventual 
use  by  the  fleet.  This  takes  into  account  both  the  technical  problems  that  must  be 
solved  to  successfully  achieve  the  research  objectives,  and  the  difficulties  associated 
with  bridging  the  gap  between  6.3  and  6.4,  as  well  as  any  special  burdens  associated 
with  fielding  the  capability.  We  evaluated  transition  probability  in  terms  of  the  severity 
of  the  remaining  technical  problems  to  accomplish  the  research  objectives,  the  qual¬ 
ity  of  the  transition  plan,  and  the  experience  of  the  project  team  with  developing  and 
fielding  similar  technologies. 

Value  Descriptions  and  Scales 

In  the  prior  application  of  this  decision  framework  to  industrial  materials  research, 
RAND  used  anchored  scales  that  specified  the  threshold  requirements  for  assigning 
specific  values.  For  example,  on  a  scale  from  0  to  1,  the  requirement  for  assigning  a  1 
for  performance  potential  was  that:  “Experimental  data  and  technical  literature  indi¬ 
cate  that  the  desired  materials  properties  have  been  achieved  by  the  candidate  material 
under  all  necessary  conditions ,”3  Thresholds  for  other  values  on  the  scale  were  defined 
according  to  combinations  of  the  level  of  achievement  of  the  suite  of  materials  prop¬ 
erties  and  the  conditions  under  which  they  were  achieved.  For  example,  0.75  corre¬ 
sponded  to  partial  achievement  under  most  conditions. 


3  Silberglitt  and  Sherry,  2002,  Table  3.2,  p.  26. 
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In  the  current  effort,  RAND  both  clarified  and  simplified  the  use  of  the  anchored 
scales  by:  (1)  separating  the  factors  that  entered  into  the  scale  value  descriptions  (level 
of  achievement  and  conditions  in  the  above  example),  (2)  asking  explicit  questions  to 
determine  each  one,  and  (3)  defining  scales  based  on  all  possible  combinations  of  the 
answers.  This  allowed  the  evaluators  to  focus  separately  on  each  aspect  of  value  and 
to  consider  each  aspect  of  the  R&D  project  independently.  For  capability,  the  aspects 
considered  were  importance  to  the  island  defense  scenario  and  pervasiveness  of  impact 
across  important  scenarios.  For  performance  potential,  the  aspects  were  achievement  of 
specified  level  of  performance  and  extent  to  which  these  were  achieved  under  relevant 
scenario  conditions.  For  transition  probability,  the  aspects  were  extent  of  remaining 
technical  problems,  experience  fielding  similar  technology,  and  quality  of  the  transi¬ 
tion  plan.  The  actual  value  descriptions  and  scales  used  in  the  case  study  are  shown  in 
Tables  3.1  through  3.6  and  Appendix  C. 


Table  3.1 

Capability  _ _ _ _ _ _ 

The  capability  that  the  project  is  aimed  at  achieving  will  be  evaluated  based  on  its 
importance,  at  a  specified  level  of  performance,  to  a  specific  warfighting  scenario, 
as  well  as  the  extent  to  which  it  influences  other  important  scenarios. 

Assuming  the  project  is  fully  successful,  the  resulting  capability  would  be: 

□  Critical  to  success  in  the  scenario 

□  A  major  factor  for  success  in  the  scenario 

□  Helpful  to  success  in  the  scenario 

□  Not  relevant  to  or  possibly  detrimental  to  success  in  the  scenario 

How  would  you  assess  the  applicability  of  this  resulting  capability  across  impor¬ 
tant  scenarios? 

□  Pervasive  across  many  scenarios 

□  Useful  in  a  number  of  different  scenarios 

□  Applicable  to  a  very  limited  number  of  scenarios  similar  to  this  one _ 
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Table  3.2 

Performance  Potential 

Hie  performance  potential  will  be  evaluated  based  on  the  extent  to  which  the  proj¬ 
ect  may  provide  performance  consistent  with  achieving  the  required  capability. 

Assuming  the  project  is  fully  successful,  the  performance  needed  to  achieve  the 
required  capability  for  the  scenario  would  be: 

□  Fully  achieved 

□  Partially  achieved 

□  Hardly  achieved  at  all 

Assuming  that  the  project  is  fully  successful,  the  performance  described  above 
would  be  achieved  under: 

□  All  relevant  scenario  conditions 

□  Most  relevant  scenario  conditions 

□  Some  relevant  scenario  conditions 

□  A  limited  number  or  none  of  the  relevant  scenario  conditions 


Table  3.3 

Transition  Probability 

The  transition  probability  will  be  evaluated  based  on  the  quality  of  the  transition 
plan  and  the  difficulty  of  remaining  technical  and  fielding  problems. 

The  project  and  project  team  is  presendy  characterized  as: 

□  No  remaining  technical  problems;  experience  fielding  similar  technology 

□  Remaining  technical  problems;  experience  fielding  similar  technology 

□  No  remaining  technical  problems;  no  experience  fielding  similar  technology 

□  Remaining  technical  problems;  no  experience  fielding  similar  technology 

The  transition  plan  for  this  project  is: 

□  Well  conceived  and  appears  to  be  implementable 

□  Has  some  problems  with  cost,  schedule,  or  other  fielding  burdens 

□  Has  major  problems  with  cost,  schedule,  or  other  fielding  burdens 

□  Is  severely  flawed  or  nonexistent 
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Table  3.4 
Capability  Scale 


Pervasive 

Number  of  Different 

Limited  Number 

Critical 

5 

4 

3 

Major  Factor 

4.5 

3.5 

2.5 

Helpful 

4 

3 

2 

Not  Relevant 

3 

2.5 

1 

Table  3.5 

Performance  Potential  Scale 

Fully 

Partially 

Hardly  At  All 

All 

5 

4 

2.5 

Most 

4 

3 

2 

Some 

3 

2.5 

1 

Limited  or  None 

2 

1 

0 

Table  3.6 

Transition  Probability  Scale 

No  Technical  Problems 

No  Experience  Fielding 

Technical  Problems 

No  Technical  Problems 

or 

and 

and 

Technical  Problems 

No  Experience 

Experience  Fielding 

Experience  Fielding 

Fielding 

Implementable 

5 

4 

3 

Some  Problems 

4 

3 

2 

Major  Problems 

3 

2 

1 

Severely  Flawed 

2 

1 

0 

Data  Requirements 

Effective  use  of  the  anchored  scales  requires  accurate  and  sufficient  data.  In  this  effort, 
RAND  used  the  scenario  definition  shown  in  Appendix  B,  together  with  the  com¬ 
bined  expertise  of  an  expert  panel  of  senior  naval  officers  and  civilian  executives  for 
the  capability  evaluation.  For  performance  potential  and  transition  probability,  the 
RAND  project  staff  compiled  summaries  of  each  of  the  20  ONR  Code  31  6.2  research 
projects  evaluated  in  the  case  study.  These  summaries  were  based  on  proposal  informa¬ 
tion  provided  by  ONR,  and  were  compiled  in  a  consistent  format  for  each  project  that 
included  an  overview,  and  all  specific  information  relevant  to  each  of  the  questions  in 
Appendix  C,  each  in  a  separate  section  with  appropriate  title.  The  project  summary 
format  is  shown  in  Table  3.7  and  Appendix  D. 

In  some  cases,  there  was  insufficient  data  in  proposals  to  compile  necessary  sec¬ 
tions  of  the  summary.  In  these  cases,  RAND  staff  contacted  the  Principal  Investigators 
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Table  3.7 

Project  Summary  Format 

Project  Title 


Overview 

A  brief  summary  of  the  project  background  and  objectives,  including: 

-  The  goal  for  the  operational  capability  of  the  project 

-  Who  is  going  to  use  it  or  benefit  from  it 

-  What  is  being  proposed 

Capability 

Description  of  the  operational  capability  that  is  being  sought  through  R&D,  who 
is  going  to  use  it,  and  why  it  is  needed. 

Performance  Potential 

Description  of  the  level  of  performance  that  is  being  sought  through  R&D,  and 
the  conditions  under  which  the  performance  will  be  achieved.  For  example: 

-  Savings  in  some  combination  of  time  (e.g.,  time-to-attack),  quality  (e.g.,  of 
data),  and  cost  (e.g.,  operational  cost,  acquisition  cost,  maintenance  cost) 

-  Improvement  over  performance  of  an  existing  system  by  specific  amount 

Transition  Probability 

Technical  Approach :  Description  of  the  technical  approach  to  achieving  the 
objectives,  including  discussion  of  the  current  state-of-the-art  and  how  it  will  be 
improved  by  the  R&D  and  the  current  status  of  the  research. 

Experience  Fielding:  Description  of  any  experience  of  members  of  the  project 
team  with  fielding  similar  capabilities  or  technologies. 

Transition  Plan :  Description  of  the  intended  transition  targets  and  the  plan  for 
accomplishing  transition  of  the  capability  to  development  and  fielding,  including 
current  status. 


of  the  projects  to  request  additional  information,  which  was  included  in  the  sum¬ 
mary. 

We  note  that  the  case  study  evaluation,  like  any  evaluation  using  anchored  scales, 
was  limited  by  the  available  data.  As  noted  in  Chapter  4,  some  members  of  the  expert 
panel  commented  that  they  thought  the  island  defense  scenario  was  too  limited  in  its 
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scope  of  required  capabilities.  As  stated  above,  RAND  used  a  second  aspect,  perva¬ 
siveness  across  important  scenarios,  in  the  anchored  scale  for  capability  in  an  effort 
to  counteract  such  a  problem.  However,  in  future  uses  of  the  decision  framework  by 
ONR,  we  recommend  that  specific  scenarios  (e.g.,  Office  of  the  Secretary  of  Defense 
(OSD) /Joint  Chiefs  of  Staff  (JCS)-approved  scenarios)  or  other  means  of  defining 
warfighting  requirements  be  defined  by  the  Navy,  in  order  to  ensure  that  capability 
valuation  is  in  accordance  with  current  priorities. 

Members  of  the  expert  panel  also  commented  that  they  needed  more  information 
on  the  R&D  projects  than  was  available  in  the  project  summaries  in  order  to  accurately 
answer  the  anchored  scale  questions.  In  future  uses  of  this  decision  framework  by 
ONR,  we  recommend  that  the  Program  Managers  or  the  Principal  Investigators  of  the 
projects  compile  the  summaries,  based  on  specific  data  requirements.  To  ensure  that 
sufficient  data  are  available  for  the  evaluation,  ONR  could  provide  the  summaries  in 
draft  form  to  the  evaluators  prior  to  Round  1  of  the  evaluation,  allow  them  to  ask  for 
additional  data  where  desired,  and  then  have  revised  summaries  meeting  these  require¬ 
ments  prepared  before  Round  1 .  This  approach  requires  a  commitment  of  time  and 
effort  by  ONR  and  the  research  performers,  but  would  ensure  that  the  evaluation  is 
based  on  the  best  available  data  on  the  R&D  projects.4 

It  is  important  to  keep  in  mind  that  the  evaluation  described  in  this  report  is  only 
a  case  study  application  of  the  RAND  PortMan  R&D  portfolio  management  decision 
framework.  As  such,  it  was  performed  within  the  time  and  budget  constraints  of  the 
project  and  should  not  be  regarded  as  a  complete  or  accurate  evaluation  of  the  20  sam¬ 
ple  R&D  projects  evaluated.  It  is,  however,  valid  as  a  case  study  demonstration  of  the 
use  of  the  framework,  because  all  of  the  evaluators  had  exactly  the  same  data  on  which 
to  base  their  evaluations,  and  the  RAND  E-DEL+I  consensus  generation  method  was 
consistently  applied  using  the  decision  frameworks  anchored  scales,  together  with  a 
representative  and  well-qualified  expert  panel. 


Overview  of  the  RAND  E-DEL+I  Technique  and  Process 

E-DEL+I  is  a  collaborative  consensus-building  method  that  combines  hardware,  soft¬ 
ware,  networking,  and  communications  capabilities  with  interlinking  mathematical 
processes  to  effectively  create  collaborative  working  environments  in  which  experts 
in  diverse  locations  can  virtually  work  side  by  side  to  accomplish  phases  of  research. 
E-DEL+I  exercises  can  be  conducted  in  two  modes:  real-time  and  non-real-time 
modes.  In  the  real-time  mode  used  in  this  work,  a  multi-round  consensus-building 


4  Among  the  lessons  learned  in  the  DoN  Fiscal  Year  2000  6.3  Review  (Kostoff  et  al.,  2001)  was  that  a  successful 
R&D  evaluation  requires  a  high-level  commitment  of  the  organization  to,  among  other  things,  allocate  the  time 
and  effort  necessary  to  prepare  adequate  data. 
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exercise  can  be  completed  in  as  little  as  two  hours  with  only  basic  business  tools  such 
as  a  telephone,  email,  and  a  standard  software  package  such  as  Excel. 

Comparison  of  E-DEL+I  and  Traditional  Consensus  Building  Methods 

E-DEL+I  retains  the  positive  aspects  of  traditional  consensus-building  methods,  while 
avoiding  the  principal  drawbacks  associated  with  the  traditional  techniques.  One  tra¬ 
ditional  primary  method  is  to  conduct  live  discussions  among  expert  panel  members. 
This  method  favors  participants  with  the  strongest  oral  communication  skills.  In  addi¬ 
tion,  implementations  of  the  discussion  method  generally  involve  bringing  together 
experts  to  a  single  location,  a  task  that  requires  considerable  logistical  coordination  and 
can  be  expensive. 

The  other  primary  method  involves  written  exercises.  For  example,  the  traditional 
Delphi  method  (Dalkey,  1969)  is  a  technique  for  conducting  consensus  building  exer¬ 
cises  in  writing.  The  hallmarks  of  the  traditional  Delphi  method  are  that  the  partici¬ 
pants  are  anonymous,  and  that  the  assessments  are  made  independently.5  Its  drawbacks 
are  that  it  favors  participants  with  strong  writing  skills,  it  does  not  permit  the  benefits 
of  live  discussions  among  the  experts  to  be  captured,  and  it  can  be  very  time  consum¬ 
ing.  Such  an  approach  can  be  costly  to  implement,  and  follow  up  is  invariably  required 
to  maximize  participation. 

E-DEL+I  allows  for  written  arguments  and  responses,  as  well  as  live  discussions 
among  exercise  participants,  while  preserving  anonymity  and  independence.  In  addi¬ 
tion,  E-DEL+I  allows  participants  to  be  physically  located  in  diverse  locations  during 
the  exercise.  A  typical  real-time  portion  of  an  E-DEL+I  exercise  can  be  completed  in 
the  time  span  of  a  business  meeting  (two  to  three  hours).  Complete  four  round  exer¬ 
cises  using  E-DEL+I  can  be  accomplished  in  as  little  as  a  day,  extended  to  a  few  weeks, 
or  designed  for  any  desired  time  span.  Hence  completing  an  E-DEL+I  exercise  can 
take  only  a  fraction  of  the  time  that  traditional  methods  require.  In  addition,  since  the 
exercises  exploit  commonly  existing  electronic  capabilities  and  infrastructure,  with  no 
traveling  expenses,  E-DEL+I  exercises  are  typically  far  less  expensive  to  complete  than 
comparable  traditional  exercises. 

The  E-DEL+I  Process 

The  E-DEL+I  process,  consisting  of  four  rounds,  is  schematically  represented  in  Figure 
3.2.  In  the  first  round,  experts  supply  assessments  in  response  to  a  questionnaire  that 
presents  the  research  questions.  For  the  case  study  application  of  the  RAND  Port- 


5  A  number  of  organizations  have  successfully  used  Delphi  techniques  for  planning  in  the  face  of  uncertainty. 
The  Graduate  School  of  Business  at  the  University  of  Southern  California  (USC)  regularly  conducts  Delphi-like 
exercises  that  involve  several  dozen  experts.  Japan  and  Germany  use  Delphi  approaches  to  determine  consensus  of 
their  national  experts  on  future  trends  in  science  and  technology  for  the  purpose  of  strategically  orienting  Japanese 
and  German  economies  towards  future  markets  (NISTEP,  1994).  The  French  Bureau  d’Economie  Theorique  et 
Appliquee  and  the  Laboratorire  ^Intelligence  Organisationnelle  used  a  Delphi-like  study  to  derive  explanatory 
trajectories  to  fight  diseases  such  as  cancer  and  Alzheimers  disease  (Ronde,  2001). 
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Figure  3.2 

E-DEL+I  Consensus  Formation  Process 
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Man  decision  framework,  the  research  questions  were  the  anchored  scale  questions 
presented  in  Tables  3.1  to  3.3  and  Appendix  C.  The  first  round  assessments  are  based 
on  the  individual  specialized  knowledge  of  each  expert,  using  the  data  provided  in 
the  background  material,  which  in  this  case  was  the  scenario  shown  in  Appendix  B, 
together  with  the  project  summaries  compiled  by  RAND,  in  the  format  shown  in 
Table  3.7  and  Appendix  D. 

The  input  for  the  second  round  is  a  statistical  summary  of  the  first-round  inputs, 
along  with  written  justifications  of  the  round  one  assessments  that  were  provided  by 
the  experts.  This  is  forwarded  to  each  participant,  along  with  a  blank  (second  round) 
questionnaire.  The  experts  review  the  feedback  material  and  make  a  second  assessment, 
this  time  supplying  arguments  for  any  first-round  minority  positions. 

In  the  case  study,  the  first  two  rounds  were  conducted  electronically,  but  not  in 
real  time.  The  questionnaire  and  background  materials  were  sent  to  the  experts,  and 
their  responses  received,  using  email,  over  a  period  of  about  one  week.  This  allowed 
the  participants  the  opportunity  to  review  the  background  material  and  familiarize 
themselves  with  the  questionnaire  format. 

Rounds  3  and  4  were  conducted  in  real  time,  and  were  preceded  by  a  facilitated 
telephone  conference  discussion  that  was  focused  on  areas  in  which  consensus  had 
not  been  reached  in  the  first  two  rounds.  The  facilitator  s  role  was  to  direct  the  discus¬ 
sions  and  to  ensure  that  all  viewpoints  were  heard.  Following  this  discussion  period, 
the  E-DEL+I  process  calls  for  a  statistical  summary  and  minority  arguments  from  the 
second  round  to  be  provided  to  the  experts,  along  with  a  blank  questionnaire  for  the 
third  round.  In  the  third  round,  participants  again  provide  assessments  and  defenses 
for  minority  positions,  this  time  after  reviewing  the  feedback  material  and  considering 
what  was  presented  during  the  discussion  period.  A  second  real-time  discussion  period 
then  focuses  on  remaining  unresolved  issues.  The  summary  and  minority  opinions, 
along  with  a  fourth  questionnaire  are  sent  to  the  participants  at  the  end  of  this  second 
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discussion  period.  In  the  fourth  round,  the  experts  provide  their  final  assessments, 
after  reviewing  and  considering  all  of  the  feedback  material,  as  well  as  insights  they 
may  have  gained  from  the  discussion  periods.  The  final  consensus  positions  are  derived 
from  statistical  analysis  of  this  final  assessment,  which,  like  each  of  the  previous  ones, 
the  experts  complete  independently. 

Previous  RAND  Experience  Using  the  E-DEL+I  Method 

RAND  has  used  E-DEL+I  to  complete  a  phase  of  research  for  four  Army  projects.  In 
each  case,  the  research  question  involved  experts  in  a  number  of  Army  areas.  Two  of  the 
exercises  were  real  time,  another  was  not  real  time,  and  the  fourth  was  a  hybrid. 


Army  Basic  Research  Technologies  This  real-time  E-DEL+I  exercise  sought  to  determine 
optimal  outsourcing  approaches  for  Army  Basic  Research  (6.1)  areas.  Fourteen 
participants  assessed  twelve  Basic  Research  areas  with  respect  to  five  outsourcing 
approaches.  At  the  end  of  the  fourth  round,  consensus  was  reached  on  eleven  of  the 
twelve  areas  (92percent)  (Wong,  2003). 


Army  Applied  Research  Technologies  This  real-time  E-DEL+I  exercise  sought 
to  determine  optimal  outsourcing  approaches  for  Army  Applied  Research  (6.2) 
technologies.  Fourteen  participants  assessed  twenty  Applied  Research  technologies 
with  respect  to  five  outsourcing  approaches.  At  the  conclusion  of  the  fourth  round, 
a  consensus  was  reached  on  eighteen  of  the  twenty  technologies  (90percent)  (Wong, 
2003). 


Assessment  of  Organizational  Structure  Alternatives  for  Army  Research  This  non- 
real-time  exercise  sought  to  assess  fifteen  organizational  structure  alternatives  on  how 
well  the  structures  are  suited  for  accomplishing  eight  critical  Army  research  functions. 
Many  of  the  organizational  structure  alternatives  were  innovative  concepts  not  yet 
in  practice.  We  judged  this  effort,  with  120  responses  requested  per  round,  to  be  too 
large  for  a  real-time  exercise.  At  the  end  of  the  fourth  round,  consensus  was  reached  on 
93percent  of  the  120  points. 


Alternative  Armies  This  project  used  an  E-DEL+I  exercise  to  refine  approaches  to 
Army  strategic  direction.  Twenty-four  experts  assessed  six  alternative  strategic  directions 
against  sixteen  criteria.  A  hybrid  approach  was  used  for  this  exercise  whereby  the 
participants  participated  in  three  real-time  discussions  and  a  mandatory  preparatory 
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session,  but  completed  the  questionnaires  over  a  series  of  four  two-day  periods.  By  the 
end  of  the  fourth  round,  the  experts  reached  consensus  on  96  percent  of  the  points. 

We  will  see  in  Chapter  4  that  the  case  study  of  the  RAND  PortMan  decision 
framework  involved  five  to  seven  experts  who  assessed  twenty  Navy  Applied  Research 
(6.2)  projects  with  respect  to  six  dimensions.  At  the  end  of  the  fourth  round,  consensus 
was  reached  on  91  percent  of  the  issues. 


CHAPTER  FOUR 

Case  Study  Description 


This  chapter  describes  the  case  study  evaluation  of  20  sample  ONR  Code  31  6.2 
applied  research  projects  that  RAND  performed  using  its  E-DEL+I  method  to  apply 
its  PortMan  decision  framework.  This  case  study  was  performed  between  January  19 
and  January  30,  2004. 


Composition  of  the  Expert  Panel 

The  case  study  evaluation  was  performed  by  an  expert  panel  of  senior  Naval  officers 
and  civilian  executives.  The  panel  was  drawn  from  a  list  of  candidates  compiled  by 
senior  RAND  and  ONR  staff,  with  assistance  from  USNR  officers.  The  criteria  for 
panelists  was  breadth  and  depth  of  experience  that  would  provide  expertise  to  answer 
the  questions  in  Tables  3.1  to  3.3  and  Appendix  C  for  specific  ONR  applied  research 
projects,  as  well  as  the  absence  of  conflicts  of  interest  with  respect  to  these  projects. 
RAND  also  sought  a  balance  of  organizational  and  technological  perspectives  in  order 
to  avoid  overall  bias.  Twelve  individuals  initially  agreed  to  serve  on  this  panel,  but  a 
number  of  these  found  that  they  were  unable  to  meet  the  required  commitment.  Seven 
panel  members  completed  the  first  round  evaluation,  and  five  of  these  completed  all 
four  rounds.  The  composition  of  the  panel  (organization  represented  and  location)  is 
shown  in  Table  4.1. 


The  Sample  ONR  Code  31  6.2  Research  Projects 

The  projects  to  be  evaluated  were  chosen  from  a  group  of  proposals  provided  to  RAND 
by  ONR  Code  31  management.  All  were  6.2  proposals  in  either  the  Command,  Con¬ 
trol  and  Combat  Systems  (C2CS)  or  Communications  subunits  of  Code  31  and  were 
categorized  within  the  early  stage  D&I  part  of  6.2.  Projects  were  combined  if  they  were 
strongly  related,  and  the  primary  selection  criterion  was  the  amount  of  data  available 
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Table  4.1 

Representation  of  Navy  E-DEL+I  Panel  of  Experts 


Representation 

Location 

Naval  Postgraduate  School 

Monterey,  California 

Operational  Liaison  Defense  Advanced  Research 

Arlington,  Virginia 

Projects  Agency  (DARPA) 

NETWARCOM 

Norfolk,  Virginia 

Naval  War  College 

Newport,  Rhode  Island 

Navy  Systems  Interoperability  Center 

San  Diego,  California 

Plans  and  Programs  Office  of  Defense  Research 

Washington,  D.C. 

and  Engineering  (OSD-ATL) 

Expeditionary  Warfare  Division  (N75)  in  OPNAV 

Washington,  D.C. 

in  the  proposals.  We  note  that  these  projects  are  a  small  fraction  of  Code  31  and  are 
not  necessarily  representative  as  a  group  of  either  Code  31  or  its  D&I  component.  In 
some  cases,  RAND  staff  called  the  Principal  Investigators  in  order  to  request  additional 
data  that  would  make  the  project  suitable  for  the  evaluation.  Appendix  E  gives  a  brief 
overview  of  each  project,  using  generic  project  titles.  The  actual  project  titles  and  the 
project  summaries  in  the  format  of  Table  3.7  and  Appendix  D  are  not  provided  in 
this  report  for  reasons  of  confidentiality,  but  are  available  with  proper  authorization 
through  request  of  the  sponsor.  Table  4.2  shows  the  generic  project  titles  and  abbrevia¬ 
tions  for  the  projects  used  in  the  analysis  described  in  Chapter  5. 


The  Assessment  Dimensions 

The  expert  panel  members  individually  assessed  each  project  with  respect  to  six  dimen¬ 
sions.  These  dimensions  were  the  questions  shown  in  the  value  descriptions  of  the 
RAND  PortMan  decision  framework,  as  adapted  for  ONR  (Tables  3.1  to  3.3  and 
Appendix  C).  The  dimensions  for  capability  were  relevance  (to  the  island  defense  sce¬ 
nario  show  in  Appendix  B)  and  applicability  (across  a  range  of  important  warfighting 
scenarios).  The  dimensions  for  performance  potential  were  achievement  (of  required 
performance  level)  and  conditions  (relevant  to  warfighting  scenarios).  The  dimen¬ 
sions  for  transition  probability  were  status  (judged  in  terms  of  remaining  technical 
problems  and  experience  of  the  project  team  with  fielding  similar  technologies)  and 
implementability  (judged  in  terms  of  quality  of  the  transition  plan).  The  experts  evalu¬ 
ated  each  project  by  selecting  from  the  list  of  responses  shown  in  Tables  3.1  to  3.3 
and  Appendix  C  the  one  that  they  believed  best  completed  the  statement  for  that 
dimension  and  that  project.  Their  responses  were  based  on  their  expertise  and  the  data 
provided,  which  consisted  of  the  scenario  description  (Appendix  B)  and  the  project 
summaries  (in  the  format  shown  in  Table  3.7  and  Appendix  D).  They  were  also  asked 
to  provide  comments  that  explained  their  choices  of  responses. 
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Table  4.2 

Case  Study  Project  Titles  and  Abbreviations 


Generic  Project  Titles  _ _ _ Abbreviations 

Alternative  Human-Machine  Interface  Project  1  A1 

Alternative  Human-Machine  Interface  Project  2  A2 

Alternative  Human-Machine  Interface  Project  3  A3 

Distributed  Sensor  Fleet  Project  1  D1 

Distributed  Sensor  Fleet  Project  2  D2 

Expeditionary  Comms  Infrastructure  Project  1  EH 

Expeditionary  Comms  Infrastructure  Project  2  EI2 

Expeditionary  Comms  Infrastructure  Project  3  EI3 

Expeditionary  Comms  Infrastructure  Project  4  EI4 

Expeditionary  Comms  Upgrade  Project  1  EU1 

Expeditionary  Comms  Upgrade  Project  2  EU2 

Network/information  Infrastructure  Project  1  N1 

Network/Information  Infrastructure  Project  2  N2 

Optimization  or  Decision-Aid  Project  1  01 

Optimization  or  Decision-Aid  Project  2  02 

Optimization  or  Decision-Aid  Project  3  03 

Optimization  or  Decision-Aid  Project  4  04 

Optimization  or  Decision-Aid  Project  5  05 

Optimization  or  Decision-Aid  Project  6  06 

UAV  General  Research  Project  1  U1 


The  Assessment  Questionnaire 

The  assessment  questionnaire  was  an  Excel  workbook  consisting  of  1 1  worksheets,  as 
described  in  the  following.  Six  of  the  worksheets  included  assessment  forms  for  one  of 
the  six  assessment  dimensions.  The  other  worksheets  provided  information,  instruc¬ 
tions,  summaries  of  the  assessments  completed  so  far,  and,  after  the  initial  round, 
statistical  feedback  on  the  responses  of  other  experts. 

General  Info 

This  worksheet  listed  the  events  that  would  occur  during  the  exercise.  It  described  the 
material  each  expert  would  receive  for  each  round,  the  actions  they  were  expected  to 
take  in  each  round,  and  the  due  dates  of  each  round  s  response. 

Relevance 

This  worksheet  contained  the  definition  of  the  Capability/Relevance  dimension  and 
the  assessment  form  the  experts  would  complete  to  record  their  assessments  of  the  20 
projects  with  respect  to  Capability/Relevance. 
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Applicability 

This  worksheet  contained  the  definition  of  the  Capability/Applicability  dimension  and 
the  assessment  form  the  experts  would  complete  to  record  their  assessments  of  the  20 
projects  with  respect  to  Capability/Applicability. 

Achievement 

This  worksheet  contained  the  definition  of  the  Performance  Potential/Achievement 
dimension  and  the  assessment  form  the  experts  would  complete  to  record  their  assess¬ 
ments  of  the  20  projects  with  respect  to  Performance  Potential/Achievement. 

Conditions 

This  worksheet  contained  the  definition  of  the  Performance  Potential/Conditions 
dimension  and  the  assessment  form  the  experts  would  complete  to  record  their  assess¬ 
ments  of  the  20  projects  with  respect  to  Performance  Potential/Conditions. 

Status 

This  worksheet  contained  the  definition  of  the  Transition  Probability/Status  dimen¬ 
sion  and  the  assessment  form  the  experts  would  complete  to  record  their  assessments 
of  the  20  projects  with  respect  to  Transition  Probability/Status. 

Implementability 

This  worksheet  contained  the  definition  of  the  Transition  Probability/Implementability 
dimension  and  the  assessment  form  the  experts  would  complete  to  record  their  assess¬ 
ments  of  the  20  projects  with  respect  to  Transition  Probability/Implementability. 

Summary 

This  worksheet  automatically  recorded  the  experts  assessments.  The  experts  could  use 
this  worksheet  to  see  an  overview  of  their  assessments  and  to  identify  which  assess¬ 
ments  they  had  not  yet  made.  It  included  the  instruction:  “No  entries  should  be  made 
on  this  worksheet.” 

Reference 

This  worksheet  included  the  project  summaries  compiled  by  RAND  in  the  format 
of  Table  3.7  and  Appendix  D,  one  for  each  of  the  20  ONR  Code  31  6.2  projects,  as 
well  as  the  description  of  the  island  defense  scenario  (Appendix  B).  It  also  included  a 
directory  with  links  that  facilitated  navigation  within  this  worksheet  to  review  specific 
projects. 

Feedback 

This  worksheet  provided  a  summary  overview  of  the  statistical  modes  for  the  previous 
round  $  responses.  This  worksheet  was  not  included  in  the  Round  1  packages. 
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Rd  X  Text 

This  worksheet  contained  the  (anonymous)  supporting  arguments  and  written  general 
comments  submitted  with  the  previous  round  s  responses.  This  worksheet  was  not 
included  in  the  Round  1  packages. 

In  order  to  make  the  workbook  more  user-friendly,  each  of  the  six  assessment 
worksheets  contained  directions,  links  to  the  project  summaries  and  the  scenario, 
drop-down  menus  with  the  assessment  response  choices,  the  statistical  feedback  of 
the  previous  round  for  that  dimension,  and  a  link  to  the  supporting  arguments  of  the 
previous  round.  An  example  of  the  Round  2  Capability/Relevance  worksheet  is  shown 
in  Figure  4.1. 


Dynamics  of  the  Evaluation 

There  were  seven  participants  in  Round  1  of  the  exercise,  which  was  accomplished  via 
email  from  January  19-21,  2004.  Using  a  simple  majority  consensus  standard,  under 
which  consensus  is  declared  if  more  than  half  of  the  participants  (i.e.,  4  out  of  7  for  this 
panel)  support  a  particular  position,  59  of  120  (6  x  20)  project-dimension  pairs  (49 
percent)  reached  consensus.  This  is  higher  than  comparable  to  past  exercises,  in  which 
approximately  30  percent  show  a  consensus  position  in  the  first  round. 

Figure  4.1 

Screenshot  from  E-DEL+I  Excel  Workbook 
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All  seven  participants  completed  Round  2,  which  was  accomplished  via  email 
from  January  23-26,  2004.  Using  the  same  simple  majority  consensus  standard,  104 
of  120  project-dimension  pairs  (87  percent)  reached  consensus.  This  dramatic  rise 
is  typical  of  Round  2  results,  although  the  magnitude  of  consensus  is  significantly 
higher  than  comparable  to  past  exercises,  in  which  approximately  50  percent  show  a 
consensus  position  in  the  second  round.  We  believe  that  the  dramatic  rise  in  Round  2 
consensus  is  due  to  “normalization”  of  interpretations  that  the  experts  gain  by  review¬ 
ing  the  statistical  feedback  and  reading  the  justifications  of  their  colleagues.  Some  of 
the  increase  may  also  be  attributed  to  changed  assessments  motivated  by  the  written 
feedback  material. 

After  the  second  round,  ten  projects  showed  consensus  in  all  six  dimensions.  Of 
the  remaining  ten  projects,  the  least  understood  dimension  was  Performance  Poten¬ 
tial/Conditions.  RAND  prioritized  the  issues  to  be  discussed  during  the  teleconference 
held  on  January  30,  2004  based  on  the  number  of  nonconsensus  dimensions  remain¬ 
ing  for  each  project.  During  the  first  discussion  period,  which  was  held  at  the  begin¬ 
ning  of  the  teleconference,  and  in  which  five  of  the  experts  participated,  discussions 
were  held  on  the  six  projects  that  had  more  than  one  nonconsensus  dimension. 

Round  3  was  completed  by  the  five  participants  during  a  break  in  the  telecon¬ 
ference  after  the  first  discussion  period.  Using  the  same  simple  majority  consensus 
standard  (three  out  of  five  for  Round  3),  111  of  120  project-dimension  pairs  (93 
percent)  reached  consensus.  This  increase  is  less  than  typical  Round  3  results,  in  which 
improvements  average  20  to  30  percentage  points.  However,  the  Round  2  results  were 
exceptionally  high,  so  in  absolute  terms,  the  Round  3  consensus  percentage  was  within 
the  typical  Round  3  range.  More  significantly,  Round  3  showed  a  consensus  for  eight 
of  the  eleven  project-dimension  pairs  that  were  discussed  during  the  first  discussion 
session.  This  indicates  that  the  discussions  were  of  value  and  that  the  exchange  helped 
the  experts  come  to  agreement  on  73  percent  of  the  points  discussed.  The  discussions 
also  resulted  in  other  changes  of  position,  hence  only  seven  more  project-dimension 
pairs  reached  majority  consensus  in  Round  3  as  compared  to  Round  2.  We  note,  also, 
that  since  the  number  of  participants  dropped  from  seven  in  Round  2  to  five  in  Round 
3,  the  Round  3  results  used  a  different  majority  standard  to  determine  consensus.  After 
the  third  round,  a  total  of  eleven  projects  showed  consensus  in  all  six  dimensions.  Of 
the  remaining  nine  projects,  the  least  understood  dimension  was  still  Performance 
Potential/Conditions. 

Expert  Panel  Concerns  and  Final  Assessments 

After  the  participants  completed  and  submitted  their  Round  3  responses,  the  tele¬ 
conference  resumed  with  a  second  discussion  session.  During  this  session,  individual 
expert  panel  members  were  allowed  to  raise  general  issues  concerning  the  exercise.  The 
two  issues  that  most  concerned  the  panel  were  the  choice  of  scenario  and  the  absence 
of  cost  and  schedule  information  in  the  project  summaries. 
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Concerns  with  respect  to  the  scenario  were  that  it  was  too  limited  in  terms  of 
both  missions  and  applicable  capabilities.  As  noted  previously,  the  dimension  Capa¬ 
bility/Applicability  was  intended  to  counterbalance  this  potential  problem  by  basing 
the  evaluation  on  all  scenarios  that  the  expert  believed  important.  Nevertheless,  the 
RAND  project  team  concluded  that  in  future  applications  of  this  framework,  the  deci¬ 
sion  on  which  scenario(s)  to  use  to  value  capabilities  should  be  made  by  the  sponsor. 

The  second  concern  was  that  the  experts  found  it  to  be  exceedingly  difficult  to 
make  a  sound  judgment  on  the  Transition  Probability/Implementability  dimension 
without  cost  and  schedule  information  on  the  projects.  Since  this  information  was 
absent  because  it  had  not  been  available  in  a  uniform  enough  fashion  in  the  proposals, 
one  way  to  solve  this  problem  would  be  to  require  the  Principal  Investigators  to  pro¬ 
vide  these  data  for  the  evaluation. 

Overall,  members  of  the  expert  panel  expressed  satisfaction  with  the  format  of  the 
exercise  and  found  it  to  have  been  useful  and  insightful.  Most  stated  that  they  learned 
something  from  their  participation. 

All  five  participants  completed  Round  4  following  the  teleconference,  and  sub¬ 
mitted  their  final  assessments  to  RAND  via  email  within  a  few  days.  Using  the  same 
simple  majority  consensus  standard,  109  of  120  project-dimension  pairs  (91  percent) 
reached  consensus.  This  small  drop  in  consensus  is  not  typical,  but  has  been  seen  in 
other  exercises.  It  shows  that  two  Round  3  consensus  positions  were  quite  weak.  For 
the  fourth  and  final  round,  the  experts  were  to  give  final  assessments  considering  all 
of  the  feedback  material  and  exchanges  made  during  the  discussions,  as  if  each  of 
them  were  the  final  decisionmaker.  Under  these  conditions,  some  experts  changed 
their  positions,  indicating  that  they  were  not  thoroughly  convinced  of  the  Round  3 
assessments.  Still,  in  absolute  terms,  the  Round  4  consensus  percentage  was  within  the 
typical  Round  4  range. 

A  total  of  eleven  projects  showed  consensus  in  all  six  dimensions.  Of  the  remain¬ 
ing  nine  projects,  the  least  understood  dimension  was  again  Performance  Potential/ 
Conditions.  Eight  project-dimension  pairs  showed  double  modes  (i.e.,  two  different 
positions  tied  with  the  most  participants  choosing  them)  in  the  final  assessments.  This 
is  higher  than  typical,  but  not  surprising  because  of  the  relatively  small  number  of 
participants.  Only  three  of  the  double  modes  indicate  bipolar  splits.  The  other  double 
modes  were  in  adjacent  positions,  so  small  differences  of  interpretation  can  account 
for  these  results.  Figure  4.2  shows  an  example  of  the  modes  (assessment  positions  with 
the  highest  frequency  of  responses)  of  the  frequency  distributions  of  the  Rounds  1—4 
responses  for  Capability/Relevance.  The  shaded  cells  show  where  majority  consensus 
positions  were  achieved. 

Summary  of  Consensus  Positions 

This  case  study  evaluation  exhibited  several  consensus-building  patterns,  ranging  from 
a  consensus  in  the  first  round  that  held  through  the  last  round,  to  double  modes  and 
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Figure  4.2 

Modes  and  Consensus  Positions  for  Capability/Relevance 


Project 

Designator 

Project 

Relevance  Modes 

Round  1 

Round  2 

Round  3 

Round  4 

A1 

Alternative  Human-Machine  Interface  Project  1 

Helpful 

Helpful 

Helpful 

Helpful 

A2 

Alternative  Human-Machine  Interface  Project  2 

Helpful 

Helpful 

Helpful 

Helpful 

A3 

Alternative  Human-Machine  Interface  Project  3 

Helpful 

Helpful 

Helpful 

Helpful 

D1 

Distributed  Sensor  Fleet  Project  1 

Helpful 

Helpful 

Helpful 

Helpful 

D2 

Distributed  Sensor  Fleet  Project  2 

Helpful 

Not  relevant 

Helpful 

Not  relevant 

Helpful 

Not  relevant 

Helpful 

Critcal 

Ell 

Expeditionary  Comms  Infrastructure  Project  1 

Helpful 

EI2 

Expeditionary  Comms  Infrastructure  Project  2 

Not  relevant 

Not  relevant 

Helpful 

Helpful 

EI3 

Expeditionary  Comms  Infrastructure  Project  3 

Helpful 

Major  factor 

Helpful 

Major  factor 

Helpful 

Major  factor 

Helpful 

Major  factor 

EI4 

Expeditionary  Comms  Infrastructure  Project  4 

Helpful 

EU1 

Expeditionary  Comms  Upgrade  Project  1 

Helpful 

Helpful 

Helpful 

Helpful 

Helpful 

Major  factor 

Helpful 

Major  factor 

EU2 

Expeditionary  Comms  Upqrade  Project  2 

Helpful 

N1 

Network/Information  Infrastructure  Project  1 

Helpful 

Helpful 

Helpful 

Major  factor 

N2 

Network/Information  Infrastructure  Project  2 

Helpful 

Helpful 

Helpful 

Helpful 

01 

Optimization  or  Decision-Aid  Project  1 

Helpful 

Helpful 

Helpful 

Helpful 

02 

Optimization  or  Decision-Aid  Project  2 

Helpful 

Helpful 

Helpful 

Helpful 

03 

Optimization  or  Decision-Aid  Project  3 

Not  relevant 

Helpful 

Not  relevant 

Not  relevant 

Not  relevant 

Helpful 

Helpful 

Helpful 

04 

Optimization  or  Decision-Aid  Project  4 

Not  relevant 

05 

Optimization  or  Decision-Aid  Project  5 

Helpful 

Helpful 

Helpful 

Helpful 

06 

Optimization  or  Decision-Aid  Project  6 

Helpful 

Helpful 

Helpful 

Helpful 

U1 

UAV  General  Research  Project  1 

Helpful 

Not  relevant 

Helpful 

Helpful 

Helpful 
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no  consensus  throughout  the  four  rounds.  Although  there  was  the  unusual  occur¬ 
rence  of  a  drop  in  the  number  of  participants  in  Round  3,  the  overall  pattern  towards 
consensus  was  not  disturbed.  Table  4.3  shows  a  round-by-round  summary  of  the  con¬ 
sensus  positions  of  the  expert  panelists  for  each  of  the  six  assessment  dimensions.  The 
entries  for  each  assessment  dimension  are  the  number  of  projects  for  which  there  was  a 
simple  majority  consensus  for  that  round,  with  a  possible  maximum  of  20.  The  total  is 
the  sum  of  the  consensus  positions  for  each  project,  with  a  possible  maximum  of  120. 
The  percentage  is  the  percentage  of  project/assessment  pairs  for  which  consensus  was 
achieved.  After  Round  4,  five  of  the  six  assessment  dimensions  showed  a  majority  con¬ 
sensus  in  at  least  18  of  the  20  projects,  with  the  dimensions  Capability/Applicability 
and  Transition  Probability/Status  showing  consensus  for  all  20  projects.  This  suggests 
that,  despite  the  problems  with  data  and  scenarios  discussed  above,  the  experts  were 
able  to  agree  on  most  of  the  assessment  dimensions. 

The  one  dimension  in  which  there  was  substantial  lack  of  consensus  was  Perfor¬ 
mance  Potential/Conditions,  for  which  only  14  of  20  projects  showed  consensus.  We 
note  that  on  the  other  Performance  Potential  dimension — Achievement,  consensus 
was  reached  for  19  of  the  20  projects.  So  the  experts  did  agree  on  the  ability  to  achieve 
the  required  performance  levels,  but  not  on  whether  or  not  they  could  be  achieved 
under  conditions  relevant  to  the  island  defense  scenario  and  other  important  scenarios. 
This  may  reflect  the  difficulty  of  specifying  the  exact  conditions  of  use  and,  especially, 
the  need  for  detailed  performance-level  research  objectives  under  specific  conditions 
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Table  4.3 

Summary  of  Consensus  Positions  of  the  Expert  Panelists 


Number  of  Occurrences  of  Majority  Consensus 


Relevance 

Applicabitiy 

Achievement 

Conditions 

Status 

Implementability 

Total 

Percentage 

Round  1 

10 

12 

10 

7 

8 

12 

59 

49% 

Round  2 

19 

20 

17 

12 

18 

18 

104 

87% 

Round  3 

19 

20 

19 

15 

20 

18 

111 

93% 

Round  4 

18 

20 

19 

14 

20 

18 

109 

91% 

in  order  to  properly  evaluate  applied  research  proposals.  In  future  uses  of  this  decision 
framework,  such  detailed  data  will  be  sought,  together  with  more  detailed  descriptions 
of  required  capabilities  and  operational  conditions  for  the  scenario  (s)  used  in  the  evalu¬ 
ation. 

As  we  shall  see  in  Chapter  5,  the  level  of  consensus  reached  for  the  120  project- 
dimension  pairs  was  sufficient  to  determine  the  three  components  of  EV  and  then 
perform  a  portfolio  analysis  of  the  20  projects,  including  explicitly  the  uncertainty  in 
capability,  performance  potential,  and  transition  probability  defined  by  the  spread  of 
the  expert  assessments. 


CHAPTER  FIVE 

Analysis  of  the  Case  Study  Results 


This  chapter  describes  RAND’s  analysis  of  the  case  study  results.  We  present  the  analy¬ 
sis  in  three  parts.  The  first  section  shows  the  results  for  EV  and  compares  the  projects 
with  respect  to  the  ranges  of  EV  into  which  they  fall.  The  next  section  discusses  the 
results  for  the  three  components  of  EV — capability,  performance  potential,  and  transi¬ 
tion  probability.  This  section  also  explains  and  provides  some  examples  of  our  method 
for  displaying  uncertainty.  The  final  section  uses  the  case  study  results  to  demonstrate 
how  to  use  the  RAND  PortMan  decision  framework  to  perform  a  portfolio  analysis. 
We  stress  that,  because  of  the  limitations  of  this  case  study  evaluation,  principally 
insufficient  data  and  a  scenario  of  limited  scope,  these  results  should  be  viewed  as  an 
example  and  not  an  accurate  representation  of  the  projects  that  were  evaluated. 


EV  Ranges  of  the  Projects 

The  RAND  E-DEL+I  evaluation  described  in  Chapter  4  provided  five  individual  expert 
assessments  on  six  dimensions  for  each  of  the  20  research  projects.  In  this  chapter,  we 
combine  the  results  for  the  appropriate  pairs  of  dimensions  that  determine  the  three 
components  of  the  EV  of  each  project  in  the  RAND  PortMan  decision  framework 
(capability,  performance  potenti al,  and  transition  probability),  according  to  the  scales 
shown  in  Tables  3.4  to  3.6  and  Appendix  C. 

According  to  the  RAND  PortMan  decision  framework,  the  expected  value  (EV) 
of  a  research  project  is  the  product  of  capability,  performance  potential,  and  transi¬ 
tion  probability  (Silberglitt  and  Sherry,  2002).  A  useful  interpretation  of  this  equation 
is  that  the  performance  potential  (i.e.,  the  performance  level  that  would  be  achieved 
if  the  project  were  successful),  scales  the  capability  value  that  is  based  on  achieving  a 
required  level  of  performance.  That  scaled  capability  is  then  further  reduced  by  the 
transition  probability  in  order  to  obtain  the  expected  value  of  the  research,  including 
its  subsequent  R&D  stages  and  the  fielding  of  the  resulting  process,  component,  or 
system.  To  visually  capture  these  relationships,  we  plot  in  Figure  5.1  the  product  of  the 
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Figure  5.1 

EV  Ranges  of  the  Case  Study  Projects 


Transition  probability 

RAND  MG271-5. 1 


mean  values  of  capability  and  performance  potential  versus  the  mean  values  of  transi¬ 
tion  probability. 

The  EV  of  each  project  is  shown  in  Figure  5.1  in  parentheses  next  to  its  data 
point.  EV  contours  are  also  shown,  expressed  in  percentages  of  the  maximum  EV  of 
125  (which  corresponds  to  a  value  of  5  each  for  capability,  performance  potential,  and 
transition  probability).  Figure  5.1  shows  that  the  20  projects  fall  into  four  distinct 
ranges  of  EV,  indicated  by  the  differently  shaded  bands.  Projects  EU2  and  EI2  are  in 
Band  1,  with  EV>30  percent  of  the  maximum  possible  value.  Projects  HI,  Al,  and 
Ell  are  in  Band  2,  with  EV  between  20  percent  and  30  percent  of  maximum.  Band 
3,  with  EV  between  10  percent  and  20  percent  of  maximum,  contains  nine  projects. 
Band  4,  with  EV<10  percent  of  maximum,  contains  six  projects. 

The  straight  line  in  Figure  5.1  from  the  origin  to  the  maximum  EV  point  at  the 
upper  right  hand  corner  of  the  plot  divides  each  band  into  two  different  regions.  Proj¬ 
ects  in  the  same  band  but  in  different  regions,  except  for  those  very  close  to  this  line, 
while  they  have  similar  EVs,  are  significantly  different  in  character.  For  example,  com¬ 
pare  projects  EI4  and  04.  Their  EVs  differ  by  slightly  more  than  ten  percent,  which 
is  within  our  range  of  uncertainty  (discussed  below).  However,  project  EI4  is  ranked 
third  on  capability  times  performance  potential,  while  project  04  is  ranked  fifteenth 
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out  of  twenty  on  this  measure  of  value  to  the  warfighter.  04  is  in  EV  Band  3  because 
it  is  tied  for  first  on  transition  probability,  whereas  EI4  is  in  EV  Band  3  (as  opposed 
to  Bands  1  or  2)  because  it  is  third  from  last  on  transition  probability.  This  suggests 
that  04  is  seeking  small  improvement  in  existing  capability,  while  EI4  has  potential  to 
make  large  advances  or  provide  an  important  new  capability. 

By  dividing  the  projects  into  EV  Bands  and  regions  of  these  bands,  Figure  5.1 
provides  a  straightforward  indication  of  which  projects  were  highly  valued  by  the  expert 
panel  (i.e.,  Bands  1  and  2),  and  the  risk-versus-reward  characteristics  of  the  projects. 
In  particular,  projects  that  are  far  from  and  above  the  line  are  potentially  high-payoff 
projects  that  were  judged  not  likely  to  transition.  These  require  management  attention 
to  determine  the  reason  for  the  low  transition  probability,  and  to  determine  if  and  how 
to  increase  the  transition  probability  without  affecting  in  a  negative  way  the  objectives 
and  approach  that  resulted  in  their  relatively  high  value  to  the  warfighter. 

By  looking  back  at  the  evaluation  data,  the  R&D  manager,  aided  by  the  plot  of 
Figure  5.1,  can  identify  the  projects  with  the  greatest  management  leverage,  and  decide 
how  to  apply  resources  to  improve  transition  probability  while  maintaining  value,  and 
push  as  many  projects  as  possible  into  the  upper  regions  of  the  upper  bands.  For  the 
projects  in  the  lower  bands,  the  manager  can  determine  from  the  data  why  these  proj¬ 
ects  were  relatively  low  in  value,  and  decide  whether  or  not  the  potential  payoff  can 
be  improved  (e.g.,  by  increasing  performance  level  objectives  or  changing  technical 
approach). 

The  RAND  PortMan  decision  framework  provides  a  basis  for  a  more  detailed 
portfolio  analysis  that  takes  into  account  each  of  the  components  of  the  EV  and  their 
uncertainties.  In  the  following  sections,  we  discuss  the  components  individually,  then 
return  to  the  EV  plot  and  describe  how  to  use  it  for  portfolio  analysis  and  investment 
strategy  development. 


The  Components  of  EV 

The  RAND  E-DEL+I  evaluation  described  in  Chapter  4  provided  five  individual 
expert  assessments  on  six  dimensions  for  each  of  the  20  research  projects.  Here  we  use 
the  mean  of  the  five  expert  assessments,  grouped  in  pairs  according  to  the  questions  of 
Tables  3.1  to  3.3  and  Appendix  C,  to  compare  the  EV  components.  We  then  describe 
a  method  for  explicitly  displaying  the  uncertainty,  or  spread,  around  the  mean  value. 

Capability  versus  Performance  Potential 

The  mean  values  for  capability,  as  determined  from  the  mean  values  of  its  two  assess¬ 
ment  dimensions,  Applicability  and  Relevance,  are  plotted  in  Figure  5.2  versus  the 
mean  values  for  performance  potential,  as  determined  from  the  mean  values  of  its  two 
assessment  dimensions,  Achievement  and  Conditions. 
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Figure  5.2 

Capability  versus  Performance  Potential 


Most  of  the  projects  fall  near  the  line  y  =  x  on  Figure  5.2,  indicating  that  the 
differences  between  their  capability  and  performance  potential  values  are  small.  How¬ 
ever,  there  are  a  few  projects  that  fall  significandy  above  the  line,  indicating  that  their 
capability  objectives  were  highly  valued  by  the  expert  panel,  but  their  performance 
potentials  were  judged  to  be  lower.  In  particular,  the  Ell  and  06  projects  were  the 
third  and  fourth  highest  valued  projects  for  capability,  yet  were  close  to  the  lowest 
valued  for  performance  potential.  This  suggests  that  these  projects  are  aimed  at  an 
important  capability,  but  that,  in  the  view  of  the  expert  panel,  their  technical  approach 
is  not  likely  to  deliver  the  level  of  performance  required  to  achieve  this  capability.  This 
result  raises  an  important  management  question  with  respect  to  these  projects  (i.e., 
could  an  alternative  technical  approach  have  a  better  chance  to  achieve  the  required 
performance  level?).  The  N1  project  also  falls  into  this  higher  capability  than  perfor¬ 
mance  potential  range,  but  it  is  still  highly  valued  compared  to  its  peers  on  both  scales, 
so  the  management  question  here  is  whether  or  not  an  already  strong  project  could  be 
made  even  more  valuable. 

There  are  a  few  projects  that  fall  below  the  line,  in  the  higher  performance  poten¬ 
tial  than  capability  range,  for  which  the  technical  approach  was  judged  to  be  sound, 
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Figure  5.3 

Capability  versus  Performance  Potential  with  Uncertainty 


Performance  potential 
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but  the  capability  was  ultimately  judged  to  be  less  valuable  (e.g.,  projects  04,  Ul,  Ol, 
03).  However,  none  of  these  are  as  far  from  the  line  as  the  three  projects  noted  above. 
We  will  have  more  to  say  about  some  of  these  projects,  as  well  as  those  in  the  group 
that  is  a  similar  distance  above  the  line  (e.g.,  projects  Dl,  05,  and  A3),  in  a  later  sec¬ 
tion  of  this  chapter. 

As  noted  previously,  it  is  important  to  recognize  that  there  is  uncertainty  in  these 
estimates  of  the  capability  value,  as  well  as  in  the  estimates  of  required  performance 
levels,  and  whether  or  not  the  proposed  technical  approach  is  capable  of  meeting  them. 
The  spread  in  the  choices  of  the  expert  panel  is  an  indication  of  this  uncertainty.  If  the 
expert  panel  had  been  a  larger  group,  we  might  represent  their  spread  of  choices  as  a 
standard  deviation  about  the  mean  values  plotted  in  Figure  5.2.  However,  with  only 
five  panelists,  we  instead  chose  to  display  the  full  range  of  values,  as  shown  in  Figure 
5.3  for  projects  Ell  and  06  discussed  above. 

In  Figure  5.3,  the  range  of  expert  valuations  is  represented  by  two  perpendicular 
straight  lines  that  intersect  at  the  mean  value  with  that  project  s  identifier.  Note  that  the 
mean  value  is  not  at  the  center  of  both  lines.  This  shows  that  the  distribution  of  expert 
assessment  values  was  not  symmetric.  If  the  mean  value  is  very  close  to  one  end  of  either 
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uncertainty  range,  then  a  single  expert  was  an  outlier  compared  to  a  consensus  or  near 
consensus  of  the  others.  For  project  06,  every  one  of  the  experts  rated  capability  signifi- 
candy  higher  than  performance  potential.  For  project  Ell,  one  expert  was  an  outlier  at 
equal  values  of  capability  and  performance  potential  of  1,  while  the  other  four  were  in 
close  agreement  on  both  capability  and  performance  potential  similar  to  06. 

We  note  the  relatively  large  uncertainty  range.  We  believe  that  this  resulted  from 
two  limitations  of  this  case  study:  (1)  the  small  number  of  experts;  and  (2)  the  data 
limitations,  most  notably  the  limited  nature  of  the  island  defense  scenario  and  the 
lack  of  data  in  the  project  summaries,  especially  with  respect  to  cost  and  schedule. 
Neither  of  these  limitations  is  inherent  to  this  decision  framework.  Both  were  a  result 
of  resource  limitations  of  the  case  study.  In  future  uses  of  this  framework,  we  will  use 
a  larger  group  of  experts,  as  well  as  a  better  definition  of  capability  value  and  more 
detailed  data.  This  will  both  reduce  the  uncertainty  range,  and  allow  the  use  of  stan¬ 
dard  deviation  as  a  more  useful  measure  of  uncertainty.  The  current  method  of  simply 
showing  the  extremes  tends  to  emphasize  the  outliers.1 

Capability  versus  Transition  Probability 

The  mean  values  for  capability,  as  determined  from  the  mean  values  of  its  two  assess¬ 
ment  dimensions,  Applicability  and  Relevance,  are  plotted  in  Figure  5.4  versus  the 
mean  values  for  transition  probability,  as  determined  from  the  mean  values  of  its  two 
assessment  dimensions,  Status  and  Implementability. 

The  plot  of  Figure  5.4  can  be  conveniently  divided  into  quadrants,  as  indicated 
in  the  figure.  Quadrant  1,  which  is  populated  by  projects  EU2,  Ell,  EI2,  Al,  and  Dl, 
with  N1  near  its  border,  is  the  region  of  high  capability  value  and  high  transition  prob¬ 
ability.  These  are  the  projects  that,  if  successful,  will  provide  a  highly  valued  capability, 
and  also  were  judged  by  the  experts  to  be  most  likely  to  be  transitioned  to  use  by  the 
Navy.  Quadrant  2,  which  is  populated  by  projects  EI4,  06,  EU1,  EI3,  Ul,  05,  N2, 
D2,  A3,  and  A2,  is  the  region  of  high  capability  value  and  low  transition  probability. 
These  projects,  if  successful,  will  also  provide  a  highly  valued  capability,  but,  in  the 
experts’  judgments,  have  remaining  problems  of  either  a  technical  or  fielding  nature. 
Projects  in  this  quadrant  require  careful  attention,  because  some  of  them  may  simply 
be  at  a  maturity  level  for  which  a  transition  plan  is  premature,  or  a  transition  plan  has 
not  yet  been  adequately  developed,  while  others  may  have  serious  and  even  fundamen¬ 
tal  technical  problems  and  fielding  burdens. 

Quadrant  3,  which  is  populated  by  projects  04  and  Ol,  is  the  region  of  low 
capability  value  and  high  transition  probability.  These  are  projects  that  are  likely  to  be 
transitioned  to  provide  incremental  improvements  in  existing  capabilities.  Quadrant 
4,  which  is  populated  by  projects  02  and  03,  is  the  region  of  low  capability  value  and 


1  We  note  that  it  will  also  be  possible  to  use  standard  deviation  as  a  measure  of  uncertainty  in  the  plot  AND  note 
the  presence  of  significant  outliers. 
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Figure  5.4 

Capability  versus  Transition  Probability 
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low  transition  probability.  In  the  experts’  judgment,  these  projects  have  serious  prob¬ 
lems  with  transition,  and  if  transitioned,  would  provide  little  value. 

Figure  5.4  is  useful  as  an  indicator  of  which  projects  are  aimed  at  valuable  capa¬ 
bilities,  and  of  how  mature  or  well  conceived  their  transition  planning  is.  However,  it 
does  not  show  the  quality  of  the  technical  approach  (i.e.,  whether  or  not  the  projects 
are  aimed  at  levels  of  performance  required  to  achieve  the  capabilities).  This  is  dis¬ 
cussed  in  the  following  section. 

Performance  Potential  versus  Transition  Probability 

The  mean  values  for  performance  potential,  as  determined  from  the  mean  values  of  its 
two  assessment  dimensions,  Achievement  and  Conditions,  are  plotted  in  Figure  5.5 
versus  the  mean  values  for  transition  probability,  as  determined  from  the  mean  values 
of  its  two  assessment  dimensions,  Status  and  Implementability. 

Figure  5.5  is  divided  into  the  same  four  quadrants  as  Figure  5.4,  but  the  meaning 
of  the  quadrants  is  different.  For  example,  Quadrant  1  is  now  the  region  of  high  per - 
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Figure  5.5 

Performance  versus  Transition  Probability  (versus  Figure  5.4) 
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formance potential  and  high  transition  probability.  Projects  in  this  region  were  judged 
to  be  likely  to  achieve  the  required  performance  for  their  capability  objectives,  indepen¬ 
dently  of  whether  or  not  those  capabilities  were  highly  valued.  Thus,  this  figure  indicates 
the  strength  of  the  technical  approach  and  objectives,  together  with  the  maturity  and 
quality  of  the  transition  plan.  The  distribution  of  the  projects  by  quadrant  is  similar 
to  that  of  Figure  5.4.  Significant  changes  occur  for  projects  with  significant  differences 
between  capability  and  performance  potential,  as  noted  in  the  discussion  of  Figure  5.2. 
More  specifically,  we  note  the  following  changes  from  Figure  5.4: 

•  Projects  Ell  and  D1  moved  from  Quadrant  1  to  the  border  between  Quadrants 
1  and  3; 

•  Projects  06  and  A3  moved  from  Quadrant  2  to  the  border  between  Quadrants  2 
and  4; 

•  Project  05  moved  from  Quadrant  2  to  Quadrant  4; 

•  Project  04  moved  from  Quadrant  3  to  Quadrant  1;  and 

•  Project  Ol  moved  from  Quadrant  3  to  the  border  between  Quadrants  3  and  1. 
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A  brief  glance  at  Figure  5.2  should  be  sufficient  for  interested  readers  to  under¬ 
stand  these  movements.  We  conclude  this  discussion  by  noting  that  it  is  important  to 
combine  the  results  shown  in  Figures  5.4  and  5.5  when  evaluating  projects  for  invest¬ 
ment.  We  will  demonstrate  this  in  the  next  section,  in  which,  as  in  Figure  5.1,  we  use 
th c  product  of  capability  and  performance  potential  to  perform  a  portfolio  analysis. 


Demonstration  of  a  Portfolio  Analysis 

As  discussed  in  the  first  section  of  this  chapter,  the  performance  potential  (i.e.,  the 
performance  level  that  would  be  achieved  if  the  project  were  successful),  scales  the 
capability  value  that  is  based  on  achieving  a  required  level  of  performance.  That  scaled 
capability  is  then  further  reduced  by  the  transition  probability  in  order  to  obtain  the 
EV  of  the  research,  including  its  subsequent  R&D  stages  and  the  fielding  of  the  result¬ 
ing  process,  component,  or  system.  To  visually  capture  these  relationships,  we  plot 
in  Figure  5.6  she  product  of  the  mean  values  of  capability  and  performance  potential 
versus  the  mean  values  of  transition  probability. 


Figure  5.6 

EV  Portfolio  Analysis  by  Quadrant 
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We  divide  Figure  5.6  into  the  same  four  quadrants  as  the  previous  two  figures. 
However,  in  this  case  the  quadrants  have  a  new  meaning.  Quadrant  1  is  now  truly  the 
high  value-high  transition  probability  quadrant,  because  it  takes  into  account  both  the 
value  of  the  capability  and  the  performance  potential  (projects  in  this  quadrant  are  aimed 
at  a  highly  valued  capability),  and  their  technical  approaches  and  objectives  are  consistent 
with  achieving  the  required  performance  level  for  this  capability.  Projects  EU2  and  EI2 
fall  into  this  quadrant,  with  project  N1  near  its  border.  With  the  more  stringent  require¬ 
ments  of  this  plot,  only  project  EI4  remains  in  Quadrant  2.  Projects  Al,  Ell,  Dl,  04, 
and  Ol  fall  into  Quadrant  3,  and  all  other  projects  fall  into  Quadrant  4. 

As  in  Figure  5. 1  >  the  EV  of  each  project  is  shown  in  Figure  5.6  in  parentheses  next 
to  its  data  point,  as  well  as  the  EV  contours,  expressed  in  percentages  of  the  maximum 
EV  of  125.  As  noted  previously,  projects  with  similar  EVs  may  be  significantly  different 
in  character,  as  indicated  by  their  position  in  different  quadrants.  For  example,  com¬ 
pare  project  04  with  project  EI4.  Their  EVs  differ  by  approximately  14  percent,  yet 
project  04  is  clearly  aimed  at  incremental  improvements  in  capability  that  were  judged 
to  be  of  low  value,  while  project  EI4,  while  its  transition  plan  is  not  yet  well-formed, 
was  one  of  the  highest  valued  projects,  taking  into  account  both  capability  and  perfor¬ 
mance  potential.  Similar  arguments  could  be  made  for  projects  02  and  Ul,  which  are 
almost  equal  in  EV,  but  vasdy  different  in  capability  and  performance  potential. 

Both  the  quadrant  position  and  the  EV  provide  us  with  interesting  data  on  the 
projects  that  are  useful  for  portfolio  analysis.  Taking  both  of  these  factors  into  account, 
as  well  as  the  fact  that  these  are  mean  values,  with  uncertainty  in  their  positions,  we 
suggest  the  groupings  shown  in  Figure  5.7  as  a  useful  way  to  combine  projects  with 
similar  characteristics  from  an  investment  strategy  viewpoint. 

Group  1  are  the  projects  that  are  most  likely  to  provide  high-value  capabilities 
that  can  be  transitioned  to  use  by  the  Navy.  Group  2  are  projects  that  are  likely  to 
be  transitioned,  but  also  likely  to  provide  only  incremental  improvements  in  current 
capabilities.  Group  3  are  projects  with  the  potential  to  provide  high  value  capabilities 
but  which  are  not  likely  to  transition  without  some  changes  to  current  approaches  and 
plans.  Group  4  are  projects  that  appear  to  be  aimed  at  incremental  capabilities  and  do 
not  have  strong  transition  plans.  Group  5  are  projects  that  have  not  yet  formed  their 
transition  strategies;  one  of  these  (project  Ul)  is  aimed  at  a  highly  valued  capability, 
while  the  other  (project  D2)  is  not. 

The  project  groupings  shown  in  Figure  5.7  are  consistent  with  the  uncertainty 
indicated  by  the  spread  of  judgments  obtained  from  the  expert  panel  evaluation.  Fig¬ 
ures  5.8  through  5.12  show  the  mean  values  for  each  group,  together  with  their  uncer¬ 
tainties,  using  the  method  discussed  previously.  Comparing  these  figures,  one  sees  that 
each  group  of  projects,  together  with  its  uncertainty  ranges,  occupies  a  separate  area  of 
the  (capability  times  performance  potential)  versus  transition  probability  plot.  These 
areas  overlap  only  slightly,  indicating  that,  with  uncertainty  taken  into  account,  the 
groupings  represent  projects  with  similar  characteristics.  We  note  also,  the  fact  that 
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Figure  5.7 

EV  Portfolio  Analysis  by  Grouping 


the  uncertainty  range  of  project  EI3  (Figure  5.10)  includes  much  higher  values  on 
both  axes,  as  compared  to  those  of  projects  06  and  N2  (Figure  5.1 1),  was  the  basis  for 
assigning  project  El 3  to  Group  3  rather  than  Group  4. 

The  project  groupings  in  Figure  5.7  identify  projects  for  which  the  R&D  manager 
may  have  the  greatest  leverage.  Consider,  for  example,  the  projects  in  Group  3,  which 
are  currently  ranked  relatively  high  in  value,  and  in  the  intermediate  to  low  range  in 
transition  probability.  In  order  to  be  successful,  these  projects  will  have  to  develop  and 
implement  a  transition  plan.  We  posit  that,  absent  attention  by  the  Program  Manager, 
the  Principal  Investigators  may  fail  to  do  this  within  the  time  needed  for  a  success¬ 
ful  transition.  Or,  if  they  are  not  constrained  sufficiently  by  floors  on  performance 
objectives,  their  tendency  may  be  to  seek  out  and  pursue  the  easiest  possible  transition 
opportunities.  This  will  increase  their  transition  probability,  but  will  likely  happen  at 
the  expense  of  lowered  performance  objectives,  which  will  move  them  toward  (the 
incremental)  Group  2.  Managements  job  in  this  case  is  to  provide  the  resources  (e.g., 
staff,  funds,  facilities,  information,  coordination,  planning)  to  overcome  either  of  these 
tendencies,  and  push  the  projects  toward  Group  1,  as  opposed  to  Group  3,  main¬ 
taining  or  even  improving  the  capability  and  performance  potential  objectives,  while 
improving  the  transition  prospects. 
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Figure  5.10 

Range  of  Uncertainty  for  Group  3 
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Figure  5.11 

Range  of  Uncertainty  for  Group  4 
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Figure  5.12 

Range  of  Uncertainty  for  Group  5 


This  type  of  leverage  is  clearly  less  for  the  projects  in  Group  4,  most  of  which  are 
aimed  at  lower  valued  capabilities,  and  some  of  which  are  further  along  in  transition 
planning.  However,  opportunities  may  exist  for  specific  projects,  for  example,  project 
06,  which  Figures  5.2,  5.3,  and  5.4  show  is  aimed  at  a  highly  valued  capability,  but 
with  relatively  low  performance  potential.  If  this  project  were  restructured  so  that  it 
might  achieve  the  level  of  performance  required  to  achieve  its  capability  objectives,  it 
might  move  to  Group  3  or  even  Group  1 . 

As  noted  previously,  the  two  projects  in  Group  5  are  entirely  different.  Project  U1 
was  relatively  highly  ranked  in  both  capability  and  performance  potential,  but  has  not 
yet  taken  any  steps  toward  transition  planning.  Thus,  substantial  leverage  may  exist  to 
push  this  project  toward  Groups  3  and  1,  as  its  transition  plan  begins  to  develop.  Proj¬ 
ect  D2,  on  the  other  hand,  was  judged  to  be  low  in  all  three  EV  dimensions.  However, 
there  was  a  large  spread  in  the  experts’  judgments,  so  the  management  question  here  is 
why  did  the  experts  disagree,  and  is  there  a  higher  value  component  to  be  pursued? 

We  suggest  the  following  steps  when  using  a  plot  like  Figure  5.7  to  develop  an 
R&D  investment  strategy: 
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1.  Review  the  data,  consider  the  reasons  for  large  uncertainties  and  outliers,  and 
note  any  effects  on  project  positions; 

2.  Give  a  high  priority  to  providing  sufficient  resources  for  each  Group  1  project  to 

achieve  its  objectives;  , 

3.  Review  the  transition  plans  of  the  projects  in  Group  2  and  allocate  funds  based 
on  the  relative  importance  of  their  capabilities,  based  on  user  inputs, 

4.  Review  the  technical  approach  and  objectives  of  the  projects  in  Group  3,  then 
provide  the  resources  to  develop  and  implement  transition  plans  that  will  allow 
the  achievement  and  fielding  of  their  identified  capabilities, 

5.  Look  for  additional  opportunities  for  projects  in  Groups  4  and  5  that  have  highly 
valued  capabilities  and  for  which  the  same  approach  as  above  in  4.  can  be  pur- 

6.  Fund  remaining  Group  4  and  Group  5  projects  based  on  the  value  of  their  capa¬ 
bilities,  if  and  only  if  a  sound  case  can  be  made  for  their  moving  into  Group  2  on 
a  specific  time  frame  consistent  with  program  objectives. 

As  noted  previously,  R&D  investments  are  made  in  the  presence  of  uncertainty- 
uncertainty  in  the  value  of  capability,  uncertainty  in  the  level  of  performance  that 
the  technology  under  investigation  and  other  related  and  unrelated  technologies  may 
achieve,  and  uncertainty  associated  with  transition  and  fielding  issues.  In  light  of  these 
uncertainties,  it  is  critical  to  a  sound  R&D  investment  strategy  to  periodically  recon¬ 
sider  the  analysis  and  decisions  described  in  this  chapter,  based  on  the  latest  and  best 
updated  available  data.  In  this  sense,  the  S&T  metric  of  our  decision  framework  is 
the  position  of  each  project  on  the  chart  of  Figure  5.7,  together  with  its  uncertainty 
monitored  over  time.  As  the  framework  is  repeatedly  used,  and  the  steps  above  repeat¬ 
edly  performed,  this  metric,  and  estimates  of  its  direction  of  motion,  will  constitute  a 
dynamic  means  for  maximizing  the  leverage  of  program  funds  to  develop  and  transi¬ 
tion  highly  valued  capabilities.  We  note  that  when  using  this  metric,  it  is  crucial  to 
either  continue  with  substantially  the  same  expert  panel,  or  to  use  a  control  group  or 
perform  a  detailed  analysis  of  variation  to  ensure  that  changes  in  the  evaluations  result 
principally  from  changes  in  the  projects  or  the  requirements,  and  not  from  changes  in 
the  panel. 


CHAPTER  SIX 

Conclusions  and  Recommendations 


The  objective  of  this  project  was  twofold:  (1)  to  adapt  RAND  s  PortMan  R&D  port¬ 
folio  analysis  decision  framework,  which  was  developed  for  the  Department  of  Energy 
(DOE),  for  use  by  the  Navy;  and  (2)  to  perform  a  case  study  evaluation  of  a  group  of 
ONR  applied  research  projects  as  a  demonstration  of  the  use  of  this  framework.  As 
described  in  the  previous  chapters  of  this  report,  both  objectives  were  accomplished, 
with  the  latter  using  the  RAND  E-DEL+I  Delphi-type  consensus  building  method. 
We  draw  the  following  conclusions  from  the  work  described  in  Chapters  1-5: 

1 .  The  adaptation  of  the  RAND  PortMan  decision  framework  (Silberglitt  and  Sherry, 
2002)  from  civilian  to  military  use  has  been  accomplished,  with  “benefit”  in  the 
DOE  version  of  the  framework  converted  to  “capability”  for  the  Navy,  “potential” 
for  DOE  converted  to  “performance  potential”  for  the  Navy,  and  “probability  of 
success”  for  DOE  converted  to  “transition  probability”  for  the  Navy.  We  note  that 
in  the  version  of  this  framework  that  we  used  for  the  case  study  evaluation,  capa¬ 
bility  value  was  defined  using  a  scenario  approach.  However,  other  approaches — 
for  example,  defining  value  in  terms  of  mission  capability  packages —  are  possible 
and  consistent  with  the  framework. 

2.  The  case  study  evaluation  of  20  ONR  Code  31  applied  research  (6.2)  projects 
demonstrated  the  use  of  this  framework  to  develop  and  analyze  the  data  needed 
to  perform  an  R&D  portfolio  analysis  and  develop  an  R&D  investment  strategy, 
including  uncertainty. 

3.  This  framework  can  be  used  with  an  expert  group  determining  the  values  of  capa¬ 
bility,  performance  potential,  and  transition  probability,  including  uncertainty. 
This  was  accomplished  in  the  case  study  evaluation  using  RAND  s  E-DEL+I  con¬ 
sensus  building  method. 

4.  This  framework  is  capable  of  comparing  and  contrasting  individual  and  groups  of 
research  projects  as  a  function  of  key  management  variables  (e.g.,  their  potential 
value  and  status  and  quality  of  transition  planning). 
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5.  This  framework  provides  a  straightforward  and  logical  set  of  repeatable  steps  (as 
described  in  Chapter  5),  based  on  auditable  data  (e.g.,  developed  for  the  case 
study  as  described  in  Chapter  4),  to  determine  the  expected  value  of  research 
projects,  together  with  a  measure  of  uncertainty. 

6.  This  framework  provides  a  set  of  parameters  for  each  research  project  (capabil¬ 
ity,  performance  potential,  transition  probability)  that,  together  with  measures  of 
their  uncertainties,  form  a  metric  that  can  be  monitored  over  time  as  the  frame¬ 
work  is  applied  iteratively. 

Based  on  the  success  of  the  case  study  evaluation,  and  the  illustrations  of  its  use  for 
portfolio  analysis  and  investment  strategy  development  presented  in  Chapter  5,  we  rec¬ 
ommend  that  ONR  take  the  next  step  of  demonstrating  this  framework  on  a  specific 
project  portfolio.  This  could  take  any  one  of  several  forms.  One  possibility  might  be 
to  incorporate  the  framework  into  an  existing  project  review  process,  such  as  that  cur¬ 
rently  used  by  ONR  Code  31,  as  described  in  Chapter  2.  In  this  case,  capability  value 
scales  would  be  based  on  a  group  of  scenarios  or  mission  capability  packages  identified 
in  collaboration  with  ONR  Code  31,  the  value  determinations  would  be  made  by  the 
existing  Code  31  expert  panel,  and  the  data  for  the  evaluation  would  be  presentations 
and  background  material  prepared  by  the  Program  Managers  and  Principal  Investiga¬ 
tors.  The  panel  evaluation  could  follow  its  current  open  meeting  consensus  building 
approach,  or  use  a  Delphi-type  consensus  building  method  such  as  RAND’s  E-DEL+I 
that  was  used  in  this  project.  RAND  would  then  compile  the  data  and  develop  and 
analyze  the  portfolio  charts  shown  in  Chapter  5. 

Another  possible  implementation  of  this  decision  framework  might  be  to  evalu¬ 
ate  specific  ONR  project  portfolios  on  a  periodic  basis  by  a  standing  expert  panel 
(e.g.,  appointed  to  staggered  three-year  terms  with  one-third  of  the  panel  new  each 
year).  The  RAND  E-DEL+I  consensus  building  method  might  be  used  to  structure 
the  panels  deliberations  together  with  the  RAND  PortMan  decision  framework  used 
to  perform  a  portfolio  analysis  and  develop  an  investment  strategy,  in  a  fashion  similar 
to  that  described  in  Chapters  4  and  5. 

We  recommend  that  the  data  for  the  evaluation  be  background  material  and 
presentations  prepared  by  Program  Managers  and  Principal  Investigators,  but  it  will 
be  critical  that  these  materials  be  prepared  according  to  specific  requirements.  It  might 
also  be  useful  to  have  the  panel  review  these  materials  in  advance  of  the  evaluation,  and 
identify  data  deficiencies  that  would  be  filled  before  the  beginning  of  the  evaluation. , 
This  application  of  RAND’s  PortMan  decision  framework  using  RAND’s 
E-DEL+I  consensus-building  method  could  be  accomplished  with  a  first  round  by 
email,  and  the  additional  rounds  during  a  one-day  or  two-day  meeting  at  which  the 
data  on  capability,  performance  potential,  and  transition  probability  would  be  devel¬ 
oped  and  plotted  on  the  portfolio  chart.  The  panel  could  then  debate  both  the  posi¬ 
tion  of  the  projects  on  the  portfolio  chart  and  the  likely  direction  of  motion.  Feedback 
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could  be  provided  to  the  ONR  Program  Managers  and  Principal  Investigators  con¬ 
cerning  where  they  were  placed  on  the  chart  and  why. 

This  standing  panel  meeting  could  be  used  as  a  tool  to  redirect  funding  towards 
the  most  promising  projects,  with  the  data  and  analysis  updated  at  each  meeting.  A 
note  of  caution:  successful  R&D  projects  take  time,  so  one  would  not  expect  to  see 
wholesale  changes  in  funding.  Perhaps  projects  might  be  reviewed  on  a  yearly  basis,  and 
funding  changes  made  only  after  multiple  unsatisfactory  reviews.  Such  yearly  reviews 
could  enable  ONR  management  to  emphasize  higher-payoff  R&D,  and  to  track  prog¬ 
ress  towards  achieving  needed  performance,  as  well  as  responding  to  changing  require¬ 
ments  in  a  timely  fashion,  consistent  with  sound  R&D  management  practices. 

Whether  it  is  applied  to  a  subprogram,  program,  or  entire  class  of  projects  (e.g., 
6.2  or  6.3),  RANDs  PortMan  decision  framework  can  be  tailored  both  to  the  scope 
of  the  evaluation  and  to  the  available  resources.  Projects  can  be  evaluated  in  groups  to 
achieve  the  desired  number  of  individual  units  for  the  evaluation.  The  amount  of  data 
used  in  the  review  would  then  be  the  maximum  consistent  with  resources  available  for 
the  evaluation. 


APPENDIX  A 

DoD  Definitions  of  R&D  Categories 


Basic  Research  (6.1) 


Applied  Research  (6.2) 


Advanced  Technology 
Development  (6.3) 

Demonstration  and 
Validation  (6.4) 


Engineering  & 
Manufacturing 
Development  (6.5) 

RDT&E  Management 
Support  (6.6) 


Systematic  study  directed  toward  greater  knowledge  or 
understanding  of  the  fundamental  aspects  of  phenomena 
and/or  observable  facts  without  specific  applications 
toward  processes  or  products  in  mind. 

Systematic  study  to  gain  knowledge  or  understanding 
necessary  to  determine  the  means  by  which  a  recognized 
and  specific  need  may  be  sent. 

Includes  all  efforts  that  have  moved  into  the 
development  and  integration  of  hardware  for  field 
experiments  and  tests. 

Includes  all  efforts  necessary  to  evaluate  integrated 
technologies  in  as  realistic  an  operating  environment 
as  possible  to  assess  the  performance  or  cost  reduction 
potential  of  advanced  technology. 

Includes  those  projects  in  engineering  and 
manufacturing  development  for  service  use  but  which 
have  not  received  approval  for  full  rate  production. 

Includes  R&D  efforts  directed  toward  support  of 
installation  or  operations  required  for  general  R&D  use. 
Included  would  be  test  ranges,  military  construction, 
maintenance  support  of  laboratories,  operations  and 
maintenance  of  test  aircraft  and  ships,  and  studies  and 
analyses  in  support  of  R&D  program. 
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Operational  System  Includes  those  development  projects  in  support  of 

Development  (6.7)  development  acquisition  programs  or  upgrades  still  in 

engineering  and  manufacturing  development,  but  which 
have  received  Defense  Acquisition  Board  (DAB)  or  other 
approval  for  production,  or  for  which  production  funds 
have  been  included  in  the  DoD  budget  submission  for 
the  budget  or  subsequent  fiscal  year. 


SOURCE:  DOD  Financial  Management  Regulation  (Volume  2B,  Chapter  5) 


APPENDIX  B 

The  Island  Defense  Scenario 


S&T  Metrics  Scenario  1:  Island  Nation  Annexation 

1 .  Purpose :  This  scenario  description  supports  the  S&T  decision  process  by  providing 
a  consistent  basis  for  applying  the  decision  framework  for  evaluation  of  a  given 
technology.  The  taxonomy  of  capabilities,  criteria  for  evaluation,  and  rules  of 
engagement  for  discussion  are  covered  in  separate  documents. 

2.  Description :  This  conflict  hypothesizes  a  small  island  nation  (U.S.  ally)  facing  a  large 
hostile  neighboring  nation  (enemy)  determined  to  annex  the  island.  It  includes  a 
defensive  (cruise  and  ballistic  missile  attack)  and  an  offensive  (time-critical  target 
of  opportunity)  component  to  be  considered  separately.  The  situation  is  provided 
as  context  for  coordinated  operations  involving  individual  units  working  together 
to  achieve  the  defensive  and  offensive  objectives.  The  effect  of  network-centric 
and/or  “traditional”  coordination  and  information  sharing  between  units  is 
critical  to  the  outcome  of  the  scenario. 

3.  General  Context: 

a.  Orders  of  Battle:  Aggressor  Nation.  The  aggressor  has  asymmetric  advantages 
over  the  island  nation  in  areas  of  submarine  warfare  versus  surface  targets; 
missiles  in  terms  of  numbers  and  range;  and  electronic  warfare  versus  the  island 
nations  radars.  Taken  together,  the  aggressor  nation  has  advantages  in  achieving 
air  superiority  and  control  of  Sea  Lines  of  Communication  (SLOC)  early  in 
the  conflict.  U.SJisland  nation.  Two  U.S.  Carrier  Batde  Groups  (CVBGs)  are 
initially  operating  east  of  the  island  outside  of  enemy  reach.  Aegis  cruisers 
perform  ballistic  defense  duties  off  the  islands  two  major  ports,  and  attack 
submarines  (nuclear  propulsion)  (SSNs)  are  assigned  to  attack  aggressors 
interdiction  submarines.  U.S.  Naval  air  forces  and  command  and  control 
assets  will  help  defeat  raids  across  the  waters  between  the  aggressor  and  island 
nations. 

b.  Strategic  Objectives :  Aggressor  Nation.  Force  island  nation  to  capitulate  before 
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U.S.  intervenes.  Probable  intense  initial  attack  to  cut  island  nation  SLOCs, 
destroy  merchant  vessels  and  infrastructure  targets.  Failing  early  capitulation, 
extend  interdiction  as  a  war  of  attrition.  U.S. /island  nation.  Hold  out  through 
initial  attack  and  force  a  war  of  attrition  with  high  enough  cost  to  aggressor 
nation  to  force  end  to  conflict  under  terms  favorable  to  island  nation.  U.S.  role  is 
specifically  to  help  island  nation  improve  its  defensive  posture  against  missile  and 
other  attacks.  No  desire  for  U.S.  to  attack  enemy  territory.  Success  dependent  on 
maintaining  air  superiority  and  control  of  SLOCs,  as  well  as  forcing  aggressor 
into  a  defensive  posture  and  thereby  limiting  its  offensive  options. 

c.  Environmental  and  Logistical  factors: 

i.  This  conflict  is  set  1 0  years  in  the  future  to  provide  time  for  new  technologies 
to  develop  and  deploy. 

ii.  Most  of  this  scenario  takes  place  in  a  littoral  environment.  Weather  and 
visibility  are  not  deemed  to  be  factors. 

iii.  The  scenario  assumes  the  U.S  forces  are  no  more  than  a  one-day  sail  from 
the  island  nation  at  the  beginning  of  hostilities. 

4.  Defensive  Scenario: 

a.  Order  of  Battle:  Aggressor  nation  will  attack  using  anti-ship  cruise  missiles  (ASCM) 
and  ballistic  missiles;  U.SJisland  nation  will  defend  using  two  Aegis  cruisers. 

b.  Tactical  Objectives:  (assume  only  two  U.S.  Aegis  cruisers  involved  in  the  area 
on  friendly  side):  Aggressor  Nation.  Launch  large  volume  of  ASCMs  versus 
U.S.  and  island  nation  targets  to  gain  an  initial  tactical  advantage  and  keep 
US/island  nation  forces  on  the  defensive;  also,  launch  ballistic  missiles  against 
critical  island  nation  infrastructure  targets  to  force  early  capitulation  and  reduce 
island  nation  logistics  and  command  and  control  (C2)  capability  in  a  war  of 
attrition.  US/island  nation.  Two  Aegis  cruisers  to  defend  themselves  against 
cruise  missile  attacks  and  prevent  enemy  ballistic  missiles  from  destroying  key 
allied  infrastructure  targets. 

c.  Required  Capabilities:  In  general,  one  cruiser  will  direct  its  SPY-1  radar  to 
detect  and  track  incoming  ASCMs,  while  another  directs  its  SPY-1  radar  to 
detect  and  track  ballistic  missiles.  This  is  done  so  both  threats  are  covered  since 
this  is  not  possible  with  a  single  radar.  Within  this  broad  assumption,  the 
scenario  has  three  alternatives  for  coordinating  and  dividing  duties  between 
the  two  Aegis  cruisers:  1)  “Platform-centric”  operations — two  cruisers  act 
almost  autonomously  (e.g.,  no  mechanism  on  board  either  ship  automatically 
shares  information  on  the  arriving  threat  and/ or  firing  solution,  and  no  central 
command  authority  directs  the  defensive  response).  This  alternative  stipulates 
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that  the  ballistic  missile  defense  ship  must  prioritize  and  service  ballistic 
missile  threats,  the  cruise  missile  defense  ship  must  prioritize  and  service  cruise 
missile  threats,  and  both  ships  must  be  ready  to  decide  whether  to  switch  roles 
based  on  remaining  inventories  of  defensive  weapons;  2)  Network  Centric 
Operations — shared  common  operational  picture  (COP) — both  ships  can  see 
and  defend  against  both  threats.  An  understanding  exists  between  both  ships 
concerning  the  nature  of  the  attack.  Each  ship  trains  its  radar  exclusively  on 
one  or  the  other  threat,  but  information  on  all  threat  trajectories  and  arrival 
times  is  shared  electronically.  Both  ships  continue  to  operate  independendy, 
and  both  ships  have  cruise  missile  and  ballistic  missile  defense  responsibilities. 
Coordination  difficulties  are  likely  in  prioritizing  and  servicing  the  targets; 
3)  Network  centric  operations — Cooperative  Engagement  — both  ships  have 
access  to  complete  defense  solutions  and  the  allocations  of  ships  to  targets  is 
controlled  centrally  by  one  of  the  ships.  Both  good  connectivity  and  automated 
systems  to  support  real-time  decision  making  are  required.  Both  ships  service 
both  incoming  threats. 

i.  High-Level  Capabilities  needed  include: 

1.  Threat  axis  determination  (intel  and  intelligence,  surveillance,  and 
reconnaissance  [ISR]) 

2.  Threat  prioritization 

3.  Outer  layer  defense 

4.  Inner  layer  defense 

5.  Survivability  (for  self-defense  component) 

ii.  Enabling  capabilities  include: 

1.  Connectivity 

2.  Secure  information  sharing 

3.  Information  filtering  and  processing 

4.  Automated  decision  support  (deconfliction,  attack  criteria,  etc.) 

5.  Firing  rate  of  friendly  weapons 

6.  Sensor  Performance  Optimization 
5.  Offensive  Scenario 

a.  Order  of  Battle:  Aggressor  nation  will  face  counter  attack  against  its  Kilo-class 
submarine  by  U.S.  assets  including  a  688-class  SSN,  a  Virginia-class  SSN,  and 
an  F/A-18  armed  with  standard  land  attack  missiles  (SLAMs). 
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b.  Tactical  Objectives:  Aggressor  nation.  Deploy  a  Kilo  interdiction  submarine  to 
conduct  SLOC  operations  against  allied  surface  targets.  U.S./island  nation:  Kill 
the  Kilo  on  the  surface  as  it  emerges  from  the  harbor  using  a  Virginia-class  ISR 
submarine  to  provide  real  time  targeting  data  combined  with  an  F/A-18  armed 
with  a  SLAM-ER  missile  guided  by  global  position  system  (GPS)  and  inertial 
navigation  system  (INS)  systems,  plus  an  electro-optical  passive  seeker. 

c.  Capabilities  Required:  The  primary  decision  to  be  made  is  whether  to  dispatch  an 
F/A-18  armed  with  a  standard  land-attack  missle-exteneded  response  (SLAM- 
ER)  to  attack  the  Kilo  or  leave  it  to  the  Virginia  class  SSN,  whose  torpedoes 
are  not  optimized  for  the  water  depth  and  which  would  compromise  the  ISR 
mission.  Once  again,  three  alternatives  are  offered  with  a  range  of  connectivity 
and  coordination:  1)  Platform-centric  operations — in  this  scenario,  the  ISR 
SSN  reports  up  the  chain  of  command  to  the  operational  commander  who  then 
alerts  the  CVBGs  that  a  submarine  has  left  port.  No  direct  communications 
take  place  between  the  ISR  SSN  and  the  F/A-18.  The  Commander,  Joint  Task 
Force  (CJTF),  develops  the  attack  plan  and  places  the  F/A-18  in  an  “alert-5” 
status.  The  F/A-18  flies  out  to  its  missile  launch  point  under  operational  control 
of  the  carrier,  which  may  abort  the  mission  based  on  threat  to  the  aircraft.  The 
ISR  SSN  provides  updates  regarding  the  target  through  the  operational  chain 
of  command,  involving  multiple  human  handoffs  and  communication  system 
nodes;  2)  Network  Centric  Operations— The  ISR  SSN  has  two-way  link 
with  the  carriers  through  Link- 16,  and  also  provides  “courtesy  copies  of  its 
communications  to  the  operational  chain  of  command,  who  may  manage  by 
exception  as  they  watch  the  scenario  play  out.  The  controlling  carrier  uses  two- 
way  communications  with  the  F/A-18  to  control  its  operation  and  to  confirm 
threat  status  updates.  The  F/A-18  receives  some  updates  directly  from  the  SSN 
with  some  latency  due  to  security  considerations.  The  SSN  may  decide  based 
on  tactical  considerations  (e.g.,  submerging  target)  whether  to  let  the  F/A-18 
conduct  the  attack  or  to  carry  out  the  attack  itself;  3)  Future  network-centric 
warfare  (NCW)  Operations — An  uninhabited  combat  air  vehicle  (UCAV) 
is  launched  from  a  ship  within  the  CVBG  and  acts  as  terminal  targeter  and 
shooter.  The  ISR  submarine  takes  control  of  the  UCAV  in  the  final  portion  of 
the  mission. 

i.  The  high  level  capabilities  needed  are: 

1 .  Area  of  probability  (AOP)  determination  (ISR,  intel) 

2.  Targeting  and  attack  criteria  decision  support 

3.  Reachable  aim  points  (range  and  accuracy) 

4.  Target  neutralization 
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5.  Battle  damage  assessment 

6.  Attacker  survivability 
Enabling  capabilities  include: 

L  Connectivity 

2.  Secure  information  sharing 

3.  Information  filtering  and  processing 

4.  Automated  decision  support  (deconfliction,  attack  criteria,  etc.) 

5.  Firing  rate  of  friendly  weapons 

6.  Sensor  performance  optimization 


APPENDIX  C 

Value  Descriptions  and  Scales 


Value  of  S&T  projects  will  be  based  on  three  factors: 

Value  =  Capability  x  Performance  Potential  x  Transition  Probability 
Each  factor  will  be  estimated  based  on  answers  to  the  following  questions: 


Capability 

The  capability  that  the  project  is  aimed  at  achieving  will  be  evaluated  based  on  its 
importance,  at  a  specified  level  of  performance,  to  a  specific  warfighting  scenario,  as 
well  as  the  extent  to  which  it  influences  other  important  scenarios. 

Assuming  the  project  is  fully  successful,  the  resulting  capability  would  be: 

□  Critical  to  success  in  the  scenario 

□  A  major  factor  for  success  in  the  scenario 

□  Helpful  to  success  in  the  scenario 

□  Not  relevant  to  or  possibly  detrimental  to  success  in  the  scenario 

How  would  you  assess  the  applicability  of  this  resulting  capability  across  important 
scenarios? 

□  Pervasive  across  many  scenarios 

□  Useful  in  a  number  of  different  scenarios 

□  Applicable  to  a  very  limited  number  of  scenarios  similar  to  this  one 


Performance  Potential 

The  performance  potential  will  be  evaluated  based  on  the  extent  to  which  the  project 
may  provide  performance  consistent  with  achieving  the  required  capability. 
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Assuming  the  project  is  fully  successful,  the  performance  needed  to  achieve  the 
required  capability  for  the  scenario  would  be: 

□  Fully  achieved 

□  Partially  achieved 

□  Hardly  achieved  at  all 

Assuming  that  the  project  is  fully  successful,  the  performance  described  above 
would  be  achieved  under: 

□  All  relevant  scenario  conditions 

□  Most  relevant  scenario  conditions 

□  Some  relevant  scenario  conditions 

□  A  limited  number  or  none  of  the  relevant  scenario  conditions 


Transition  Probability 

The  transition  probability  will  be  evaluated  based  on  the  quality  of  the  transition  plan 
and  the  difficulty  of  remaining  technical  and  fielding  problems. 

The  project  and  project  team  is  presently  characterized  as: 

□  No  remaining  technical  problems;  experience  fielding  similar  technology 

□  Remaining  technical  problems;  experience  fielding  similar  technology 

□  No  remaining  technical  problems;  no  experience  fielding  similar  technology 

□  Remaining  technical  problems;  no  experience  fielding  similar  technology 

The  transition  plan  for  this  project  is: 

□  Well  conceived  and  appears  to  be  implementable 

□  Has  some  problems  with  cost,  schedule,  or  other  fielding  burdens 

□  Has  major  problems  with  cost,  schedule,  or  other  fielding  burdens 

□  Is  severely  flawed  or  nonexistent 


Capability  Scale 


Pervasive 

Number  of  Different 

Limited  Number 

Critical 

5 

4 

3 

Major  Factor 

4.5 

3.5 

2.5 

Helpful 

4 

3 

2 

Not  Relevant 

3 

2.5 

1 
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Performance  Potential  Scale 


Fully 

Hardly  At  All 

All 

5 

4 

2.5 

Most 

4 

3 

2 

Some 

3 

2.5 

1 

Limited  or  None 

2 

1 

0 

Transition  Probability  Scale 


No  Technical  Problems 
Experience  Fielding 


Implementable  5 

Some  Problems  4 

Major  Problems  3 

Severely  Flawed  2 


No  Technical  Problems  Technical  Problems  No 

No  Experience  Fielding  Experience  Fielding 

or  Technical  Problems 

Experience  Fielding  _ 

4  3 

3  2 

2  1 

1 


0 


APPENDIX  D 

Project  Summary  Format 


Project  Title 

Overview 

A  brief  summary  of  the  project  background  and  objectives,  including  the  following: 

•  the  goal  for  the  operational  capability  of  the  project; 

•  who  is  going  to  use  it  or  benefit  from  it;  and 

•  what  is  being  proposed. 

Capability 

Description  of  the  operational  capability  that  is  being  sought  through  R&D,  who  is 
going  to  use  it,  and  why  it  is  needed. 

Performance  Potential 

Description  of  the  level  of  performance  that  is  being  sought  through  R&D,  and  the 
conditions  under  which  the  performance  will  be  achieved,  for  example: 

•  savings  in  some  combination  of  time  (e.g.,  time-to-attack),  quality  (e.g.,  of  data), 
and  cost  (e.g.,  operational  cost,  acquisition  cost,  maintenance  cost);  or 

•  improvement  over  performance  of  an  existing  system  by  specific  amount. 

Transition  Probability 

Technical  Approach  Description  of  the  technical  approach  to  achieving  the  objectives, 
including  discussion  of  the  current  state-of-the-art  and  how  it  will  be  improved  by  the 
R&D,  and  the  current  status  of  the  research. 

Experience  Fielding  Description  of  any  experience  of  members  of  the  project  team 
with  fielding  similar  capabilities  or  technologies. 

Transition  Plan  Description  of  the  intended  transition  targets  and  the  plan  for 
accomplishing  transition  of  the  capability  to  development  and  fielding,  including 
current  status. 
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Project  Overviews 


Alternative  Human-Machine  Interface  Project  1  (A1) 

This  project  will  design,  develop,  and  assess  an  integrated  3D  visualization  command 
system  as  a  tactical  aid  for  the  anti-submarine  warfare  decisionmaker.  The  current  research 
focus  is  on  developing,  in  a  laboratory  environment,  an  upgraded  prototype  data  proces¬ 
sor  to  submarines’  fielded  tactical  control  systems  and  associated  sensor  databases.  Creat¬ 
ing  a  reliable  and  consistent  common  tactical  picture  across  platforms  is  a  major  challenge 
to  achieving  net-centric  “speed  of  command”  in  the  multiwarfare  environment;  decision¬ 
makers  have  traditionally  assimilated  data  from  multiple  2D  displays  and  paper  plots. 


Alternative  Human-Machine  Interface  Project  2  (A2) 

The  submarine  commander  s  access  and  ability  to  use  correct  tactical  information,  includ¬ 
ing  understanding  uncertainty  in  measurements,  is  one  of  the  major  factors  in  achieving 
success  in  the  batdespace.  However,  in  most  cases  the  visualizations  do  not  account  for 
uncertainty,  contrary  to  the  assumption  of  users.  Acoustic  detection  and  tracking  of  tar¬ 
gets  is  an  area  in  which  visualizations  could  benefit  from  uncertainty  information.  The 
accuracy  of  sonar  depends  on  uncertainty  associated  with  environmental  features.  Hav¬ 
ing  the  ability  to  represent  uncertainty  in  the  targets  depth  has  the  potential  to  convey  a 
more  complete  Common  Tactical  Picture  (CTP).  Using  this  added  knowledge,  the  warf¬ 
ighter  may  be  able  to  leverage  his  improved  understanding  into  tactical  decisions  that  are 
made  more  quickly  and  are  of  higher  quality  than  those  of  his  adversary. 


Alternative  Human-Machine  Interface  Project  3  (A3) 

Existing  visualization  systems  are  awkward  and  provide  limited  interaction  capabili¬ 
ties.  Multimodal  interaction,  allowing  speech  and  3D  gesturing  can  help  make  these 
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systems  powerful  adjuncts  to  other  command  and  control  systems.  In  addition,  this 
project  will  create  a  parameterization  scheme  to  enable  a  developer  to  create  a  multi¬ 
modal  system  for  their  desired  2D  or  3D  military  application  rapidly. 


Distributed  Sensor  Fleet  Project  1  (D1) 

This  project  is  providing  for  the  real-time  allocation  and  platform  assignment  of  radar 
resources  to  perform  detection,  track  and  missile  support  functions  for  Antiaircraft 
Warfare  (AAW)  and  Theater  Ballistic  Defense  missions.  The  ability  to  replace  hard 
coded  limitations  on  sensor  utilization  with  dynamically  adjusted  data  service  rates  and 
sensor  assignments  in  response  to  the  tactical  situation  is  expected  to  improve  single 
ship  and  coordinated  warfighting  capabilities. 


Distributed  Sensor  Fleet  Project  2  (D2) 

A  communication  system  at  a  sensor  location  that  actively  transmits  has  a  large  number 
of  constraints.  The  complete  assembly  will  be  large  and  the  power  source  for  the  system 
presents  limitations  in  endurance  and  increases  in  cost.  The  proposed  system  concept  is 
to  obtain  sensor  information  from  the  data  source  by  utilizing  passive  radio  frequency 
(RF)  tag  technology.  The  technology  allows  for  significant  cost  savings  and  reduced  size. 


Expeditionary  Communications  Infrastructure  Project  1  (Ell) 

This  project  is  a  broad-based  program  focused  on  developing  technology  applications 
to  achieve  the  potential  for  an  interoperable,  manageable,  and  secure  military  inter¬ 
network  built  overtop  various  military  and  civil  subnetworks,  based  on  current  and 
emerging  standards  and  commercial-off-the-shelf  software.  The  objective  is  to  design, 
implement  and  demonstrate  a  common  technical  architecture  for  interoperable  secure 
networks  and  intended  to  lead  towards  a  basis  for  a  robust  interoperability  specifica¬ 
tion.  Ell  is  related  with  EI2  and  EI3. 


Expeditionary  Communications  Infrastructure  Project  2  (EI2) 

Commercial  Internet  today  primarily  relies  on  Internet  Protocol  version  4  (IPv4) 
networking  protocols,  yet  a  developing  standard,  Internet  Protocol  version  6  (IPv6), 
is  seen  as  a  far  more  robust  and  capable  successor  to  the  current  technology,  but  suf¬ 
fers  significantly  from  a  lack  of  compatibility  in  supporting  current  systems.  IPv6 
addresses  a  number  of  IPv4  problem  issues  and  also  provides  new  opportunities  for 
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system  and  protocol  designers.  This  project  intends  to  carefully  look  at  the  myriad  of 
technical  and  architectural  options  for  integrating  emerging  IPv6  products  and  new 
routing  products  through  the  lens  of  the  current  Automated  Digital  Network  System 
(ADNS),  which  is  presently  based  on  IPv4.  In  addition  to  exploring  future  solutions, 
plans  are  to  address  existing  networking  problems  and  shortfalls  by  capitalizing  on 
the  enhanced  capabilities  of  IPv6  technology.  Findings  from  this  project  should  have 
some  influence  on  the  direction  and  implementation  of  a  larger  parent  program. 


Expeditionary  Communications  Infrastructure  Project  3  (EI3) 

This  project  will  support  the  development  and  implementation  of  a  mobile,  wireless, 
communication  architecture  that  will  be  capable  of  supporting  a  multinational  coali¬ 
tion  in  a  tactical  environment.  The  focus  will  be  on  integrating  emerging  mobile  net¬ 
working  technologies  into  a  demonstration  system  that  can  meet  operational  require¬ 
ments.  The  military's  future  demands  will  likely  surpass  those  of  commercial  systems 
that  rely  on  fixed,  pre-positioned  infrastructure  and  looser  quality  of  service  (QoS) 
definitions.  This  proposal  is  intended  to  provide  momentum  and  act  as  a  subtask  for 
a  significant  element  of  the  planned  Expeditionary  Communications  Infrastructure 
Project  1. 


Expeditionary  Communications  Infrastructure  Project  4  (EI4) 

All  branches  of  the  American  military  are  moving  rapidly  toward  network  centric  archi¬ 
tectures  for  voice,  video,  and  data  communications.  The  major  effort  of  this  project 
will  be  to  develop  an  asymmetric  secure  link  through  the  Joint  Tactical  Radio  System 
(JTRS)  framework  to  provide  network  access  to  vulnerable  assets. 


Expeditionary  Communications  Upgrade  Project  1  (EU1) 

Networking  protocols  will  be  developed  under  this  R&D  program  that  will  enable 
mobile,  tactical,  wireless  networking  with  directional  antennas.  This  is  an  area  where 
very  little  research  had  been  done  prior  to  the  start  of  this  ONR  program  in  Fiscal 
Year  2000  most  existing  wireless  mobile  networks  use  omni-directional  antennas.  The 
current  focus  is  on  systems  that  interface  to  Defense  Information  System  Network 
(DISN)-like  data  networks  and  protocol  suites  rather  than  systems  for  which  timing 
constraints  are  intimately  tied  to  threat-object  tracking  and  weapons  delivery  (such  as 
the  Cooperative  Engagement  Processor,  or  CEP). 


Project  Overviews  69 


Expeditionary  Communications  Upgrade  Project  2  (EU2) 

This  technology  will  offer  the  warfighter  a  significant  advantage  in  the  ultrahigh  fre¬ 
quency  (UHF)  transfer  of  video,  voice,  and  target  data  for  improved  command  and 
control,  including  but  not  limited  to  control/data  transfer  for  Unmanned  Aerial  Vehi¬ 
cle  programs. 


Network/Information  Infrastructure  Project  1  (N1) 

To  achieve  the  high  levels  of  situational  awareness  and  shared  knowledge  among  all 
elements  of  a  joint  force,  warfighters  must  be  able  to  gain  access  to  all  relevant  data 
from  heterogeneous  sources,  assess  its  credibility,  and  respond  to  the  critical  changes 
in  a  meaningful  manner.  The  project  team  proposes  to  provide  an  active  capability, 
implemented  in  the  form  of  software  tools,  to  assist  operationally  in  the  integration, 
filtering,  fusing,  validating,  and  monitoring  of  information  from  distributed  fleets  of 
heterogeneous  sources  such  as  sensors,  platforms,  and  networks. 


Network/Information  Infrastructure  Project  2  (N2) 

Future  swarms  of  network-centric  combat  systems  will  increasingly  run  unobtrusively 
and  autonomously,  shielding  Naval  operators  from  unnecessary  details,  while  commu¬ 
nicating  and  responding  to  mission-critical  information  at  an  accelerated  operational 
tempo.  In  such  an  environment,  it  becomes  very  difficult  to  predict  system/package 
configurations  or  workloads  in  advance. 

Technologies  are  thus  being  developed  to  bridge  the  gap  between  military  appli¬ 
cations  and  the  underlying  operating  systems  and  communication  software  in  order 
to  provide  the  critically-important  capability  of  reusable,  reliable  services  on  which 
network-centric  warfare  must  depend.  This  program  composites  two  related  projects. 


Optimization  or  Decision-Aid  Project  1  (Ol) 

This  tool  will  provide  a  rapid,  probabilistic  method  for  predicting  course  of  action 
development  in  time  critical  targeting.  This  tool  serves  as  a  Battlespace  Decision  Aid 
to  assist  the  mission  planner  in  assessing  potential  locations  of  mobile  threats  as  well 
as  providing  mobile  target  identification.  The  software  is  a  predictive  modeling  and 
analysis  software  tool  that  integrates  mature,  commercial-off-the-shelf  technology  with 
advanced  Navy-specific  tools  that  blend  imagery,  spatial  features,  elevation,  terrain, 
with  real-time  tactical  feeds  and  geopolitical  tendency  models.  This  tool  will  operate  as 
a  network-based  software  resource. 
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Optimization  or  Decision-Aid  Project  2  (02) 

This  project  will  investigate  the  efficacy  of  enhancing  the  ability  to  defend  against  anti¬ 
ship  cruise  missiles  and  other  airborne  threats  by  providing  simulation  tools  that  assist 
in  predicting  and  assessing  the  performance  of  AAW  systems  for  single  platforms  and 
batdegroups.  The  goal  of  this  project  is  to  help  decide  where  best  to  place  and  config¬ 
ure  sensors  and  weapon  platforms  to  maximize  coverage,  given  a  set  of  AAW-capable 
ships  and  a  task  of  defending  an  area  from  air  threats. 


Optimization  or  Decision-Aid  Project  3  (03) 

The  objective  of  this  project  is  to  create  a  computerized  planner  to:  choose  land  attack 
missions;  allocate  weapons  to  meet  the  tasking  requirements  as  closely  as  possible; 
simultaneously  consider  factors  such  as  retaining  maximal  follow-on  firing  capability 
and  leveling  missile  inventory  across  designated  platforms. 


Optimization  or  Decision-Aid  Project  4  (04) 

Scheduled  shipboard  resources  have  varying  requirements  for  accomplishing  certain 
processing  tasks.  The  inherent  problem  of  reserving  machine  service  times  for  custom¬ 
ers  (e.g.,  targets)  appearing  in  the  queue  is  a  generic  problem  that  may  be  able  to  be 
solved  using  one  scheduling  optimization  algorithm. 


Optimization  or  Decision-Aid  Project  5  (05) 

Course  of  Action  replanning,  a  complex  activity,  is  invariably  required  during  com¬ 
bat  and  could  be  supported  by  software  tools.  However,  existing  decision  aids  for 
crisis-action  planning  do  not  address  this  need.  A  planner  must  manually  review  dis¬ 
joint  information  sources  to  gather  and  synthesize  this  information,  and  identify  those 
events  that  are  critical  for  monitoring. 


Optimization  or  Decision-Aid  Project  6  (06) 

A  dynamic  approach  is  demonstrated  for  achieving  the  real-time  and  integrated  decon- 
fliction  of  scheduled  fires  while  meeting  required  on-target  ordnance  delivery  rates. 
This  decision  aid  will  provide  operators  with  a  new  awareness  of  the  impact  of  their 
fires  and  will  increase  the  safety  to  non-hostiles  in  the  theater  while  decreasing  the 
fratricide  risk.  The  deconfliction  technology  begins  to  open  the  door  for  evolving  joint 
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missile  and  air-engagement  zones  and  allows  the  greater  defense  and  strike  power  from 
smaller  forces. 


UAV  General  Research  Project  1  (U1) 

Tactical  military  usage  of  unmanned  systems  for  operations  is  anticipated  to  become 
increasingly  prevalent  and  influential  in  future  naval  operations.  This  emerging  appli¬ 
cation  has  both  high  visibility  and  high  uncertainty  due  to  the  larger-scale  reliance  on 
networked  performance  and  abstraction  on  the  part  of  the  warfighters.  The  military 
application  of  unmanned  systems  to  Littoral  Warfare  is  a  natural  extension  of  a  much 
larger  technological  trend  in  automation,  standardization,  and  modularization.  This 
project  effectively  functions  as  a  portfolio  of  related  projects  with  a  number  of  tasks 
under  direct  management  and  as  well  oversees  related  work. 
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